Technology Blog

I am a dress designer and so am always on the look out for new things that I can do on my collections that will make them appealing to my customers. A customer came in with a specific request that she wanted Swarovski crystals on her dress. I was wondering where I could buy some Swarovski hot fix which can be easily fixed on dresses without any hassles. That was when I came across crystals2love.com. They sell wholesale rhinestones in different shapes and colors and they were very affordable. The delivery was also very prompt which helped me to deliver the dress on time to my customer. 

I am a person who has a hectic work schedule. Sometimes I have to work even on weekends. This greatly restricts my time for socializing as I do not get to meet people apart from my business acquaintances. So going on a date is definitely a difficult task for me. Very rarely when I go out to parties with my friends I meet some new peoples. But it becomes difficult to keep up the contacts I make there. Sometime back I came across Northern Ireland Dating which has greatly improves my social life. I have found people with whom I have common interests and went on a few dates.

Problem management is an area that needs to be clearly defined and documented. It is vital that all administration staff know who, where and when to call if problems arise. It is common to have several groups inside an organization responsible for different parts of the data warehouse. There may be a central helpdesk for application, tool or PC problems, while operations are responsible for the hardware and a separate DBA team is responsible for the database. Everyone from the users through to the developers and operators needs to know who to call.

This whole area of problem management needs to be bottomed out early in the data warehouse project. The environment and the applieation will be complex, and everyone involved in supporting the system or supplying a help service will need to be identified and trained.

Starting up and shutting down of the server, database or the data warehouse application are tasks that are likely to be performed infrequently. They are, however, tasks that it is important to get right, because shutting down the machine or database incorrectly can cause problems on restart. Each will need scripted and menu-driven procedures for shutting it down, and starting it up. The data warehouse application has daemons that need to be shut down. These should be shut down gracefully, allowing them to complete whatever work they are currently doing, not just aborted. It is useful to have options that allow a daemon to continue working, but not to accept any more work. This allows the daemons to clear their current load without taking any more on. They can then be shut down cleanly. There will be times when the daemons are required to be shut down instantly. To allow for this eventuality the warehouse manager must design the daemons to be restartable, picking up the threads of the tasks they were performing when they were crashed. At the very least the daemons should be able to restart any jobs they were running.

Logins for each user will need to be created, maintained, and deleted. This is a task that should be menu driven to prevent accidents. Adding a user with the wrorig privileges can prove to be a costly mistake. The users should all fall into easy categories or groups, allowing a template login to be created for each group. This template should have all the default accesses, profiles, roles and so on for that group. The new user can then be set up with that template as a basis and any specific requirements added.

Extra care needs to be taken with the design of privileged user accounts. The amount of damage that can be caused by the inadvertent dropping of a table or data file is enormous in a database of this size. Where possible, all privileged operations should be performed via menus that have the relevant checks in them, to prevent such accidents. A great deal of monitoring and logging is required to keep a data warehouse system ticking over correctly. Log files, which log ongoing events, and trace files, which dunip information on specific processes, will generate a vast amount of data, and can occupy a lot of disk space. Although small in comparison with the size of the database, this disk space is still significant. Scripts will be needed to maintain the logging directories. Old log files should be archived if they contain useful historical data. They should also be purged regularly, to keep space usage under control.

There are always housekeeping tasks that need to be run. Scripts that purge down directories, log off idle connections and so on are as much a part of a data warehouse as of any other system. The data feeds may arrive at any time of the day. Having some of the data arrive during the day is not an issue unless it causes network problems. You may even be able to get it loaded into the database, or at least into a staging area, if the processing cost is not too high. Data extracts are subsets of data that are offloaded from the server machine onto other machines. Extracts can be unscheduled user extracts of some query results, or they can be scheduled extracts such as data mart refreshes.

Extraction of user query results can impose a significant load on the network, and has to be controlled. Ideally, the extraction of anything larger than a few megabytes should be controlled via the schedule manager. As this may require the schedule manager to integrate with several network protocols it is not a simple task, and may require considerable programming to automate. Often the best approach is to push the extract under queue control to a file server that is on the same network segment as the user. One must note there may be security issues with this approach, and care needs to be taken not to make information available inadvertently.

The warehouse application managers are likely to have a number of service or background processes running. These processes are often called daemons. These daemons are used to log events, act as servers and so on. The monitoring of these daemons is an operational task. Ideally, there will be a menu interface that will allow the operators to start up, halt, shut down and otherwise interface with the application daemons.

Query management is a vital part of the daily operations. Some of the monitoring and control can be automated, but there will still be a requirement for a DBA to be available to deal with any problems. This is particularly important ifuser profiles are not being used. The profiles can be used to control resource usage and to force-terminate a job that has exceeded its quota. If profiles are not being used, jobs will need to be killed manually if they are using excessive resource.

Backups are usually avoided during the user day, to avoid contentions with the users’ queries. In addition, in a data warehouse the bulk of the data is read-only, and there is little that happens during working hours that will require backing up. It is usually only the overnight load and the update of the aggregations that require backing up. The exception will be if there are areas such as users’ work areas that need saving. Sometimes these need to be backed up during the day.

The “query from hell” is the DBA’s nightmare. It is a query that occupies the entire resource of a machine and effectively runs for ever, never finishing in any time-scale that is meaningful. Naturally, it is a query to be avoided. Given the vast quantity of data in a data warehouse, such queries are all too possible to generate. The problem is that it may be impossible to tell the difference between an acceptable query that takes up to 48 hours to run and one that will run for much longer and effectively never finishes. Unless it is possible to measure exactly how far a query has progressed and how much more processing it has left to do, it will be impossible to predict when a query will end. The only way of preventing such queries is by controlling exactly how much resource a query can have.

This can be done in two ways. First, profiles can be used to limit the amount of resource a user process can use. When that amount is exceeded, the query will be terminated automatically. This will not prevent a query running, but it will stop it running for ever using vast amounts of resources. The advantages of profiles are that they are automatic, and they require no external control. The disadvantage is that they kick in only after a lot of resource has already been wasted. Second, queuing of queries can be used to control their resource usage. If all . queries must be submitted via the query manager, then the degree of parallelism and other resources of the query can be controlled. Any query that is likely to run for extended periods of time can be limited in what resource it will use. This will mean that the query will take longer to run, but it will stop a single query interfering with the other processing necessary to maintain and run the data warehouse.

Typically, organizations will experiment with the concept of data analysis and educate themselves on the value of a data warehouse, prior to determining that a data warehouse is the appropriate solution. In practice, many organizations address this by sponsoring an initial prototyping activity, which is used to further understanding of the feasibility and benefits of a data warehouse. This activity is valuable, and should be considered if this is the organization’s first exposure to the benefits of decision support information.

In some instances, the data warehouse may be the first large-scale client-server solution being implemented within the organization, and will require new skills, experiences, hardware, etc. A prototyping activity on a small scale can further the educational processes as long as the prototype addresses a clearly defined technical objective; the prototype can be thrown away once the feasibility of the concept has been shown - that is, it does not become “the first bit of the data warehouse”; the activity addresses a small subset of the eventual data content of the data warehouse and the activity timescale is non-critical that is, it is seen as a time boxed effort to come to grips with the new technologies being considered.

Unlike prototypes, working models suffer in that they have a tendency to set the expectation that they will grow to the full data warehouse. This may be inappropriate in practice, because the architecture of the model may not scale up to a full data warehouse. If the requirement is to produce an early release of part of a data warehouse, in order to deliver business benefits, we suggest that you focus on the business requirements and technical blueprint phases and understand the short and medium term requirements of the data warehouse.