Multimedia Databases

Multimedia database .Presentation

2006-08-30T15:50:00.000-07:00

Multimedia database : PPT

Seminar report

2006-04-11T08:15:00.000-07:00

here is my seminar report, images are missing as i used latex2html for it.
here is the link
http://www.cse.iitb.ac.in/~ajitb/seminar/

Types of queries

2006-04-06T18:42:00.000-07:00

Now after defining these data structure our database is ready to answer fundamental queries like:

Whole Match Queries: Given a collection of N objects O1,.., On and aquery object Q find data objects that are within distance delta from Q
Sub-pattern Match: Given a collection of N objects O1,.., On and a query(sub-) object Q and a tolerance delta identify the parts of the data objects that match the query Q.
K- Nearest Neighbor queries: Given a collection of N objects O1,.., On and a query object Q find the K most similar data objects to Q.
All pairs queries (or ”spatial joins”): Given a collection of N objects O1 ,.., On find all objects that are within distance delta from each other.

for solving such queries we first need to find a distance function between two objects and find one or more numerical feature-extraction functions (to provide a quick test). Then Use a SAM (e.g., R-tree) to store and retrieve k-d feature vectors. here is an example of queries in context to computer games which uses these data structures :

Visibility - What can I see?
Ray intersections - What did the player just shoot?
Collision detection - Did the player just hit a wall?
Proximity queries - Where is the nearest power-up?

Why are apatial access methods important

2006-04-06T18:36:00.000-07:00

The main purpose is to is to support efficient spatial selection, for example range queries or nearest neighbour queries of spatial objects. Without a spatial index,every object in database need to be checked whether it meets selection criterion,i.e complete linear scan of relational database.
Clustering is needed to group those objects which are often requested together. Otherwise, many different disk pages will have to be fetched, resulting in slow response. For spatial selecting the clustering implies storing objects which are close together in reality also close together in the computer memory (instead of scattered over the whole memory).
In traditional database systems sorting (or ordering) the data is the basis for efficient searching. Higher dimensional data can not be sorted in an obvious manner, as it is possible for text strings, numbers, or dates (one-dimensional data). Basically, computer memory is one-dimensional. However, spatial data is 2D, 3D (or even higher) and must be organized somehow in the memory. An intuitive solution to organize the data is using a regular grid just as on a paper map. Each grid cell has a unique name; e.g. ’A3’, ’C6’, or ’D5’. The cells are stored in some order in the memory and can each contain a (fixed) number of object references. In a grid
cell, a reference is stored to an object whenever the object (partially) overlaps the cell. However, this will not be very efficient due to the irregular data distribution of spatial data: many cells will be empty, e.g. in the ocean, while many other cells will be overfull, e.g. in the city center. Therefore, more advanced techniques have been developed.

Representing image through quadtree

2006-04-06T18:16:00.000-07:00

Let’s say we divide the picture area into 4 sections. Those 4 sections are then further divided into 4 subsections. We continue this process, repeatedly dividing a square region by 4. We must impose a limit to the levels of division otherwise we could go on dividing the picture forever. Generally, this limit is imposed due to storage considerations or to limit processing time or due to the resolution of the output device. A pixel is the smallest subsection of the quad tree.
To summarize, a square or quadrant in the picture is either :

entirely one color
composed of 4 smaller sub-squares

To represent a picture using a quad tree, each leaf must represent a uniform area of the picture. If the picture is black and white, we only need one bit to represent the colour in each leaf; for example, 0 could mean black and 1 could mean white.

Now consider the following image :
The definition of a picture is a two-dimensional array, where the elements of the array are colored points .

Given Image in pixel form In Quad tree-form

Now when this image is represneted as quad tree it will look like :

The quad tree of the above example picture. The quadrants are shown
in counterclockwise order from the top-right quadrant. The root is the top node.
(The 2nd and 3rd quadrants are not shown.)

For matching procedure, color structure descriptor is first extracted from sample image and then matched with the descriptors associated to the images contained in the DB. Now here we can have a result image in certain range of tolerance according to two criterion : Quadtree Structure Similarity (QSS) and Quadtree Color Similarity(QCS). The main concept is that the difference in the structure of two quadtrees can be evaluated through the number of changes in the structure that need to be performed to make one of the quadtrees equivalent to the other. This process is called quadtree warping. Once the two quadtrees have the same structure, they are recursively navigated and the difference is computed between the colors of
the corresponding leaves. The final formula used for the similarity matching(SM) is the following:

SM = A1*QSS + A2*QCS

where the constants A1 and A2 are determined empirically for normalization and
weighting of the two metrics.

How is Multimedia Data different

2006-03-27T02:41:00.000-08:00

The content of multimedia data is often captured with different “capture” techniques (e.g., image processing) that may be rather unreliable. Multimedia processing techniques need to be able to handle different ways of content capture including automated ways and/or manual methods.
Queries posed by the user in multimedia databases often cannot come back with a textual answer. Rather, the answer to a query may be a complex multimedia presentation that the user can browse at his/her leisure. Our framework shows how queries to multimedia databases may be used to generate multimedia presentations that satisfy users queries-a factor that is unique to our framework.
Multimedia data is large and affects the storage , retrieval and transmission of multimedia data.
In case of video and audio databases time to retrieve information may be critical ex(Video on demand).
Automatic feature extraction and Indexing: In conventional databases user explicitly submits the attribute values of objects inserted into the database. In contrast, advanced tools such as image processing and pattern recognition tools for images, to extract the various features and content of multimedia objects. As size of data is very large we need special data structures for storing and indexing.

Types of Multimedia Data

2006-03-26T17:22:00.000-08:00

There are number of data types that can be characterized as multimedia data types. These are typically the elements or the building blocks of ore generalized multimedia environments, platforms, or integrating tools. The basic types can be described as follows :

Text : The form in which the text can be stored can vary greatly. In addition to ASCII based files, text is typically stored in processor files, spreadsheets, databases and annotations on more general multimedia objects. With availability and proliferation of GUIs, text fonts the job of storing text is becoming complex allowing special effects(color, shades..).
Images : There is great variance in the quality and size of storage for still images. Digitalized images are sequence of pixels that represents a region in the user's graphical display. The space overhead for still images varies on the basis of resolution, size, complexity, and compression scheme used to store image. The popular image formats are jpg, png, bmp, tiff.
Audio : An increasingly popular datatype being integrated in most of applications is Audio. Its quite space intensive. One minute of sound can take up to 2-3 Mbs of space. Several techniques are used to compress it in suitable format.
Video : One on the most space consuming multimedia data type is digitalized video. The digitalized videos are stored as sequence of frames. Depending upon its resolution and size a single frame can consume upto 1 MB. Also to have realistic video playback, the transmission, compression, and decompression of digitalized require continuous transfer rate.
Graphic Objects: These consists of special data structures used to define 2D & 3D shapes through which we can define multimedia objects. These includes various formats used by image, video editing applications.

Multimedia Indexing

2006-03-13T12:39:00.000-08:00

The Basic Steps for multimedia indexing is :

Provide a measure for the distance between two multimedia objects :
So we need to define a distance function D().Given two objects OA, OB the distance (=dis-similarity) of the two objects is denoted by D(OA, OB). It can be Euclidean Distance .
Similarity Queries :
- Whole query matches : for a query of object O , find the objects within the database which are with in similarity distance delta from O.
- Sub-pattern Match : Given a collection of N objects O1,…, ON and a query (sub-) object Q and a tolerance delta identify the parts of the data objects that match the query Q .
- Or we can find k nearest neigbours of object queried.
Algorithm : Sequential scanning and distance calculation with each and every object is too slow for large databases. We can use highly fine-tuned database SAMs (Spatial Access Methods) like R-trees to accelerate the search (by pruning out large portions of the database that are not promising) .

The similarity distance can be based on features extracted from image. Like in previous post for an image the similarity can be based on color, texture etc

Case Studies

2006-03-08T17:34:00.000-08:00

MediaDB is Multimedia Database Management System that creates MMDB applications to manage large volumes of multimedia objects. MediaDB is an object oriented system that supports encapsulations, inheritance hierarchies, and object management. It also supports one-to-many and many-tomany relationship between objects.

InterBase is a relational Database that has builtin support for BLOBs. It has its own interface to relational database that it manages. InterBase stores BLOBs in collection of segments. A segment in InterBase can be thought of as a fixed-length "page" or I/O block. InterBase provides special API calls to retrieve and modify segments. It allows user to open-BLOb , create-BLOB, read segmet (by get-segment) and write a segment to BLOB (put-segment).

Sybase's SQL Server supports TEXT and IMAGE data types. TEXT and IMAGE data-types can be very large up to 2GB. User can defien table as :
CREATE TABLE Page (
Number Integer ,
Content IMAGE ,
Anstract TEXT)
In terms of their internal storage representation TEXT and IMAGE column values contain pointer to firest page of multimedia column. The pages are stored as linked-list. The pages od TEXT and IMAGES colums are stored seperately fro tables of database.

XDP from Plexus is imaging database engine that provide support for BYTES and TEXT data types. A special family can be assigned to each IMAGE and TEXT column in a Table with a different storae extent or family for that tabel recors. Example :
CREATE TABEL Pages
(Number Intege,
PageImage IMAGE IN ImageFamily)
IN CompoundDocumentFamily

XDP supports hierarchial storage system, managing magnet disks, optical disks, optical jukeboxes with online, near-line (stored in the optical jukeboxes but not munted), and offline stored on a shelf) devices.

MultiMedia Datatypes in UniSQL/X Being an object relationa; database management system , UNiSQL supports most of SQL constructs. FOr multimedai data types UniSqL/X supports a class hierarchiay rooted at generalized large object (GLO) class. This class serves as the root od multimedia data types and provides a number of builot-in attributes and methods for instatiating and inheriting clients.For content of GLO objetcs user can create either Large Object (LO) or a File Based Object(FBO) . An interestin gfeature is that recovery is supported for FBOs but consisteny is not supported as external application can aloso access it. LOs are similar to BLOBs in relational databases. FBOs are stored in host file system. Two other subclasses are Audio and Image class. Audio serve as root class of various type of audio classes. Image class is the root of image types, including 2D and 3D images. It supports number of attributes and ethods(also inherited from parent classes) to support storing and manipulating data.

Content Based Retrival

2006-03-04T09:18:00.000-08:00

Content-based retrieval (CBR)of a multimedia data is retrieval that uses the text of the document rather than any added metadata. Free text searching is a good example of content-based text retrieval for text documents. The words making up the content of the document are indexed and used as the basis for retrieval, sometimes in conjunction with quite sophisticated “intelligent” software used to satisfy the query. But in case of images and videos CBR is difficult. The difficulty is that an image is an collection of pixels with color specified for each pixels but need to interpret them automatically and see meaningful regions of colour, recognise objects and identify scenes which can usefully form the basis of effective image matching processes.
How can CBR be performed for images :

Image Retrieval By Color : The colour histogram has been a simple and popular representation which captures the relative amounts of each colour in an image. But it is a global measure and does not give information about colour variations at local positions in the image. Color Histogram gives the proportio of pixels of each color within a image. Now we define a distance color set distance d^(set)=(c_q-c_t)^tA(c_q-c_t) where
Now for a query we search for minimum value of this distance.
Region Query for Images by Spatial description : Image may be divided into some regions. Here we can perform triangulation of image. Now region location can be found using Euclidean distance of centroids. When we query for an region of an image we can make the weighted sum of
as a criterion. We may also need to define spatial relation between regions for example

How can we perform CBR :
1. The application provides us a drawing tool so that we can draw image and calculate feature of query image and search accordingly in database.
2. Or we can prform Query By Example : in this case we provide an example image so that an iomage from database should be similar to it.

Using Metadata for managing Multimedia Data

2006-03-03T04:51:00.000-08:00

As in case of multimedia data the exact match paradigm for querying is no longer suitable or adequate. Also content-based processing are hard to analyse and result is not often precise. So Metadata that is derived or extracted from digital media have advantage of being more amenable to traditional data retrieval and manipulation techniques than raw digital media.

Metadata can often be stored in relatively standard relational and object oriented structures and can be easily queried though simple extension of languages such as SQL.

Type of Metadata :

Content Independent Metadata : this type of metadata captures information that does not depend on content of document with which it ios associated. Example of this type of metadata are : owner,location, modification date of document. There is no information content captured but still such metadata is useful for retrival of documents.
Content dependent Metadata : Example is size of document,number of rows and columns and othre field which depend upon content. This can be further sub-divided in two :

Direct Content Base Metadata : This is based directly on content of document. A popular example is "index for a document" .
Content Descriptive Metadata : This type of metadata describes the content of document without direct utilization of those contents.Example is "texual anonations describing an image".

Now here is how can we formally define metadata :

A media-instance is an 8-tuple
(ST, fe, ATR, A, R, F, Var1, Var2)
where:
ST is a set of objects called states, and
fe is a set of objects called features, and
ATR is a set of objects called attribute values, and
A is a map from states to sets of features, and
R is a set of relations on fe^i * ST for i > 0, and
F is a set of relations on ST, and
Var1, is a set of objects, called variables, ranging over ST, and
Var2, is a set of variables ranging over fe.

A state is smallest chunk of media data that we wish to consider. For example , in an image database each eimage may be viewed as state.
A feature is any object in a state that is of interest for multimedia database. Example for an image pic.gif showing car, the feature of pic.gif is car .
The Attributes are properties of features in a state. like for picture of car the attributes may be car color, model.
A is a feature extraction ap which tells which feature occur in which states.
R is set of relation on fe X ATR X ST --those relations are state dependent. for instance for an omage photo.gif. There may be state dependent relation ship called left_of containg tuples like (a , b , photo.gif) such that a is in left of b in photo.gif.
F is a relation on states like "before" example for such relation is (a.gif , b.gif) such that acc to before relation , photo a.gif was taken before b.gif.

Having this formal representation now we can easily formulate methods for queying database easily. Now we cab define functions like : FindObjectWithFeature(a) , this wll search database for all such states with feature containg a. Or we can list all features of an image with FindFeatures() . Other function types can be FindObjectWithFeatureandAttr(Jane,suit,blue) this query asks to find all media objects in which jane apperars in a blue suit. the above mentioned functions are available by default in SMDS-SQL. An example query is "find all images containg jane and denis , with jane to left of Denis.
SELECT M
FROM smds source1 M
WHERE FindType(M) = Image AND
M in FindObjWithFeature(Denis) AND
M in FindObjWithFeature(jane ) AND
left(jane,denis,M)

Information Management & Retrival : Introduction

2006-03-03T02:59:00.000-08:00

Basic Approches for Information management :

Conventional database system – This is the widely-used approach to manage and search for structured data. All data in a database system must conform to some predefined structures and constraints (i.e., schemas). To formulate a database query the user must specify which data objects are to be retrieved, the database tables from which they are to be extracted and predicate on which the retrieval is based. A query language for the database will generally be of the artificial kind, one with restricted syntax and vocabulary, such as SQL.
Information retrieval (IR) system – IR system is mainly used to search large text collections, in which the content of the (text) data is described by an indexer using keywords or a textual abstract, and keywords or natural language is used to express query demands. For example for an image or video we have to describe it in words or in a way need to store lot of metadata (texual form).
Content based retrieval (CBR) system – This approach is used to retrieve desired multimedia objects from a large collection on the basis of features (such as colour, texture and shape, etc.) that can be automatically extracted from the objects themselves. Although keyword can be treated as a “feature” for text data, traditional information
retrieval has much more higher performance than content-based retrieval because keyword has the proven ability to represent semantics, while no features have shown convincing semantic describing ability. But major drawback of this method is that it lacks precision.
Graph or tree pattern matching – This approach aims to retrieve object sub-graphs from an object graph according to some denoted patterns.

Advanced data structures

2006-03-01T15:33:00.000-08:00

k-dimensionl trees (k-d trees) :
Each level of a k-d tree partitions the space into two.

choose one dimension for partitioning at the root level of the tree.
choose another dimensions for partitioning in nodes at the next level and so on, cycling through the dimensions.

In each node, approximately half of the points stored in the sub-tree fall on one side and half on the other. Partitioning stops when a node has less than a given maximum number of points.

Each line in the figure (other than the outside box) corresponds to a node in the k-d tree. The maximum number of points in a leaf node has been set to 1.

Division of Space by Quadtrees :

Each node of a quadtree is associated with a rectangular region of space; the top node is associated with the entire target space.

Each non-leaf nodes divides its region into four equal sized quadrants correspondingly each such node has four child nodes corresponding to the four quadrants and so on. Leaf nodes have between zero and some fixed maximum number of points (set to 1 in example).

R trees :
R-trees are a N-dimensional extension of B+-trees, useful for indexing sets of rectangles and other polygons. Supported in many modern database systems, along with variants like R+ -trees and R*-trees.
A rectangular bounding box is associated with each tree node.

Bounding box of a leaf node is a minimum sized rectangle that contains all the rectangles/polygons associated with the leaf node.
The bounding box associated with a non-leaf node contains the bounding box associated with all its children.
Bounding box of a node serves as its key in its parent node (if any)
Bounding boxes of children of a node are allowed to overlap.

A polygon is stored only in one node, and the bounding box of the node must contain the polygon
The storage efficiency or R-trees is better than that of k-d trees or quadtrees since a polygon is stored only once.
To find data items (rectangles/polygons) intersecting (overlaps) a given query point/region, do the following, starting from the root node:

If the node is a leaf node, output the data items whose keys intersect the given query point/region.
Else, for each child of the current node whose bounding box overlaps the query point/region, recursively search the child.

SQL and Multimedia : Introduction

2006-03-01T14:08:00.000-08:00

Here is example on how to define table with large objects i.e LOB

CREATE TABLE grape
( grape_name VARCHAR2(30) ,
grape_txt CLOB DEFAULT EMPTY_CLOB() ,
picture BLOB DEFAULT EMPTY_BLOB() ,
CONSTRAINT prim_grape PRIMARY KEY (grape_name) )

here grape_text is the the description of grape and is stream of characters so is stored as CLOB. Whereas picture is stored as BLOB. Here the image will be stored in database itself. Other way is we can store it as BFILE. Then we can have used :
picture BFILE ,

Now while inserting data we have to do as :
INSERT INTO grape (grape_name, picture) VALUES ('chardonnay' , BFILENAME('PHOTO_DIR','chardonnay.jpg'))

Using DBMS_LOB Package :
CREATE PROCEDURE wine_read_bfile
IS
Lob_loc BFILE ;
Amount INTEGER :=32767
Position INTEGER := 1;
Buffer RAW(32767 ) ;
BEGIN
SELECT picture INTO Lob_loc FROM grape
WHERE grape_name = 'chardonnay' ;
DBMS_LOB.OPEN(LOB_loc, DBMS_LOBB.LOB_READONLY);
DBMS_LOB.READ(Lob_loc, Amount , Position , Buffer);
DBMS_LOB.CLOSE(LOb_loc);
END ;
here the file will be loaded in the buffer. The BFILE is manipulted through LOB locator.
You can also modify CLOB using
DBMS_LOB.WRITE(lob_loc,amount,offset, newtext ) ; here the newtext is added at position lob_loc + offset value.

Multimedia Object Size

2006-03-01T11:04:00.000-08:00

The size of multimedia data can make storage, processing, transmission and recption of data very costly.

Store data externally to database and you may just store metadata in database

Reducing size of multimedia data so that they can be stored within database

Object-relational databases offers means of stoing and processing multimedia data, based on SQL:1999. It provides two new data types that can hold media data.th first is "large object" oor LOB data type. THis has two variants, BLOB "binary large object " and CLOB Character large object. Oracle9i supports another datatype for media objects that are stored externally as operating system files (BFILE data type. LOBS in Oracle9i can hold up to 4GB size. A LOB can be broken down to chunks for processing. It also facilitates storing URLs containing audio, image, or video data stored on any HTTP server such as Oracle Application Server, Netscape Application Server, Microsoft Internet Information Server.

for more informaion here. Oracle sql synatx can be found here .

Reducing Media Object's Size

Signifiacnt reduction in media oobject size can be achieved by data compression using varios algorithms. But while doig compression we have to keep in mind that how much quality loss (in case of loosy compressions) is acceptable while decompression. In an application we can store data in one compression format and transform data in another while transmitting or presentation, so we need to store information about this. IBM's DB2 and Oracle allows us to do so by providing DB2IMAGE and ORDIMAGE types respectively. So we can store following attributes :
Compression format- such as jpeg,GIFLZW
Compression Quality value such as LOWCOMP ,HIGHCOmp
Content format value such as image type/pixel/data format
FileFormat - such as BMP, TIFF

high quality compression is obtained by using MPEG standards.

Why need Multimedia Database

2006-03-01T10:24:00.000-08:00

Data increasingly means not just numbers and small strings but multimedia data as well – structured text, images, video, audio, VR, etc.
As

multimedia data is large and will affect the storage , retrival and transmission of multimedia data.

multimedia data structures are completely different from standard database
In multimedia data structures we want content-based searching

In case of video and audio databases time to retrive information may be critical ex (Video on demand)

Multimedia Database Management System. (MMDBMS) must support multimedia data types in. addition to providing facilities for traditional DBMS like database creation, data modeling, data retrieval, data access and organization, and data independence.

Types of Multimedia datatypes :

Text (arbitary long text fields)

Still images

Graphics objects like CAD/CAM objects

Video data

Audio data

Literature Survey on multimedia database

2006-03-01T10:07:00.000-08:00

As i doing an literature survey on multimedia database for my seminar under Prof N. L. Sarda. So as go on with it i will post here what i learn.