My Database Managment System Project

Blogs to track my database managment system project. The main target of this project is learning. I am not planning to compete with SQL server - At least not now!

My Photo
Location: Cairo, Egypt

I am an Eyptian developer @ optimize software company.

Tuesday, June 21, 2005

Security (cont)

As I mentioned in my last blog, security in general have some points of difficulty. In addition to those points, I have some points of difficulty that are special to my project. These points are:
1- Working alone:

Generally, creativity is related to looking at things from more than one point of view. The problem with the human mind, is that it always looks at things from one point (or points), and it ignores all other points. This is a nature of the human mind that you cannot override, regardless of how trained and experienced you are.

So for tasks that require some sort of creativity, it is always recommended to be done by more than one person. Many persons can look at a task from many points of view and decrease the flaws in it.

Security and design are the most creative activities in the software development process, so it is always recommended to be done by more than one person. Even if you are working alone on such tasks, put your design or code, leave it for one week and then take a look at it again. This can make you forget your design or code and then read it again to make a review for your work. Of course this is not like working in groups of more than one.

For security, it is recommended that developers review each other's code (peer review) to find security holes in the code or the design. This has the following benefits:

  • Every one is unconsciously focusing on few points and ignoring other points. When another developer reviews his code, he looks at his other points, so generally security flaws decrease a lot.
  • This will create the "Hawthorne effect”. This effect is named for a factory just south of Chicago, Illinois. Researchers measured the length of time it took workers to perform tasks while under observation. They discovered that people worked faster and more effectively than they did when they weren't observed by the researchers. A Developer will work harder and try to write the most possible secure code if he knows that a peer developer will review his code. This can give him some sort of challenge.

My problem is that I am working alone on this project. I have no peers, no one to review my code, no one to think with me or design with me.

2-I have no security guy:

I am not a security expert. This is my first time to read deeply about security and of course my first security practice will contain flaws. It is recommended in any project to have a security expert (either from inside the company or form outside).

The task of the security expert is to specify security rules and recommendations for the project, review design, review code, review working plan, give training for project team in security point related to their project, etc.

Of course I am not a security expert and I cannot hire one because of the budget of the project. The budget of this project is 0$!

But of course I am not going to ignore security as a feature of the project because of the above reasons. Security education can increase the level of security of the application. Of course it will not be prefect, but it will not be 0% secure.

You cannot make an application secure without knowing what makes an application secure. Remember the word “You don't know what you don't know". This word applies to security very much.

In a good experiment, one of Microsoft security experts asked two developers of his friends to review 1000 lines of C code for security flaws. The first found 10 flaws and the second found 16.He then gave them an intense one-hour presentation about coding mistakes that lead to security vulnerabilities and how to question assumptions about the data coming into the code. Then he asked them to review the code again. The first person found another 45 flaws, and the second person found 41. Incidentally, the security expert himself had spotted only 54 flaws in the code. So the first person, who found a total of 55 flaws, had found one new flaw, and the second person, with 57 total flaws, had found the same new flaw as the first person plus two others!

Security education of the team can increase the security level of the application dramatically. For this reason, During February and March of 2002, all normal feature work on Microsoft Windows stopped. Throughout this period, the entire development team turned its attention to improving the security of the next version of the product, Windows .NET Server 2003. The goal of the Windows Security Push, as it became known, was to educate the entire team about the latest secure coding techniques, to find design and code flaws, and to improve test code and documentation.

As a result for this, the security flaws of windows server 2003 and IIS 6 was extremely less than previous windows versions.

So I am not going to ignore security or stop reading about it to increase the security level of my application although I am sure my application will have many security flaws.

Sunday, June 19, 2005

Security : The sub-feature of every feature

I continued working on requirements specification for my database engine project. The first thing that I need to specify is security features. What are they? What security features am I going to implement, and what features am I going to ignore.

My previous knowledge about security was very small. I knew some specific security subjects such as buffer overruns, some authentication, authorization and encryption techniques, but I needed a bird's eye view on the security in general. How can I make my application secure? And how can I test its security?

So I started looking for good books about this subject. I found an excellent book called "writing secure code" which was an excellent start for me to look at the security as a whole.

The first thing that I discovered is that the second statement that I wrote in this blog is wrong. As the book says, security features is different from secure features. Every feature, every function, every line of code, and every design decision, have to take security rules, threats and recommendations into account.

So, I should have written my statement like this “The first thing that i need to specify is security rules, threats and recommendations that I am going to work with in this project.

So instead of putting security as an item in the requirements documents, I have to make separate documents for security and call them security documents.

An example of a wrong development plan mentioned in the book was like this:

Milestone 0: Designs complete
Milestone 1: Add core features
Milestone 2: Add more features
Milestone 3: Add security
Milestone 4: Fix bugs
Milestone 5: Ship product

This methodology in application development is wrong for the following reasons:

  • Adding security later is wrapping security around existing features, rather than designing features with security in mind.
  • Adding any feature, including security, as an afterthought is expensive.
  • Adding security might change the way you've implemented features. This too can be expensive.
  • Adding security might change the application interface, which might break the code that has come to rely on the current interface.
  • While working on milestones 0 to 3 without security in mind, make sure that there are security bugs in the code and the design. When coming to milestone 4, you will have to rebuild the design and code that contains security bugs (probably most of the application). This will need an excess cost and time. Remember what I have said before about the exponential increase in cost of fixing bugs when discovered in the design or code stages.
In a lot of situations, developers tend to ignore security bugs to reduce cost and time, especially if the bug was discovered near the releasing time of the application. They may think of fixing it in the next releases or versions. This is extremely dangerous.

If the customers discovered that the application doesn't work as advertised, your reputation will go down, and your customers will leave you to another company.

Also fixing security bugs after production is very expensive. Microsoft security response center believes that a security bug that requires a security bulletin costs in the neighbor hood of $100000.

Fixing security bugs frequently introduces Regression Bugs to the system. Regression bugs happen when a feature works fine and after some changes it doesn't work as before. This requires a strong testing for the application after fixing security bugs.

Of course security increases cost and time, but remember the good word in software engineering: "Features, cost, schedule: choose any two".if you want to have good features for your application, then security is a must and you have to endure the excess cost and time.

Generally, security is more difficult than many items in software production for the following reasons:

1- Security is difficult to measure:

For example, if you are making a program that solves differential equations, then you can test this feature easily. Give the program a set of differential equations of different inputs and degrees. If the program gave the right answer, then this feature is working well.

You cannot do the same with security. You cannot easily say:” My application is secure, and no one can hack it or stop it". There's no 100% secure application. All what security experts trying to do, is making the process of attacking the system as much difficult as possible. The most secure application is the one that you haven't written yet. Believing in a 100% secure application is like believing in tooth fairy.

So how can I measure the security level of my application?
Some of Microsoft guys classify an application as a secure application if 12% of the time and cost were given to security.

I think this is a wrong method of measuring security. Since security is a sub-feature of every feature, it is difficult to say how much of time and cost was specified to security alone.
Also the security level required depends on the nature of the application. The level of security required for an online money transfer application is not the same as that required for a computer game.

2-The attacker always has the advantage, the defender is always in a dilemma:
This is the nature of security generally either in software or in any other field.
These are the reasons:

  • The defender must defend all points; the attacker can choose the weakest point.
  • The defender can defend only against known attacks; the attacker can probe for unknown vulnerabilities.
  • The defender must be constantly vigilant; the attacker can strike at will.
  • The defender must play by the rules; the attacker can play dirty. The defender has various well-understood white-hat tools (for example, firewalls, intrusion-detection systems, audit logs, and honey pots) to protect his system and to determine whether the system is under attack. The attacker can use any intrusive tool he can find to determine the weaknesses in the system

In addition to the general difficulties of security, I have security difficulties that are special to my application. I am going to discuss them in my next blog (God willing).

Friday, June 10, 2005

Requirements: The stage with the long lasting effect

If I was asked to summarize the science of software engineering in one statement, it would be this:
Software is not just writing code.

In small projects, you can only just write code, without an architectural design, without formal requirements, etc. But as the size of the project increases, those other things are a must.

As a developer, I feel a need to bypass this boring stage which reminds me of the governmental routine work. A lot of developers and specially managers who don't understand the importance of requirements specification may override this stage. So if you spend some time in requirements specification, you may have your manager screaming at you: "you have been working for 2 weeks and you didn't write any code?"

This phenomenon is called WISCA syndrome, which is a short form for "Why Isn't Sam Coding Anything?” and as the word tells, if this phenomenon happens, it is considered a disease in the company.

As a fact, as the size of the project increases, the stage of writing code becomes smaller compared to all stages of the project.

For example, in small sized projects as I mentioned before, the stage of code writing takes 100% of the project. For medium sized projects, it may take 60% of the project. For very big projects and systems, it may take much less than this.

In his excellent book Mythical Man-Month, Fred Brooks (on of the designers of the OS/360 operating system of IBM) says that in very big projects such as building operating systems, the estimated work becomes like this:

1/3 planning (requirements specification, design, etc).
1/6 Coding.
1/4 Module testing.
1/4 System testing.

He also says that on such big projects, the developer can only write 1000 lines of debugged code yearly.

Well, I agree with the principle, but I don't agree with this 1000 lines. Note that in the past when Fred wrote that operating system, the coding and debugging tools were not advanced as today's tools.

I consider my project as a very large project, because my previous knowledge about it are small and the number of developers working on it is small too ( I am working on this project alone).So I expect to spend 1/3 of the project time in the requirements and design stages.

In his book "Code Complete", Steve McConnell says that if fixing a requirements error discovered in the requirements stage costs 1$, it may cost 3 dollars if it was discovered in the architectural design stage, 5-10$ if discovered in the coding stage, 10$ if discovered in the system test stage, 10-100$ if discovered in the post release stage.

This means that the cost of error fixing grows exponentially through the program stages, it doesn't grow linearly, and the effect of the requirements specification stage will have an effect on all the project stages.

I started my database project with the process of requirements specification: what are the features and aspects of advanced database engines? And which of these can I implement in my project?

I wanted to know in detail the features that I am not going to implement, to put a design that may allow for adding them in the future. This is a starting point in the design process (which I didn't start yet): define what is going to change and what is not.

So I start my requirements document by writing a list of points that need to be studied. Of course as work on the project, this list will expand. These are the points that I have ( until now):
2-The query language I am going to use.
3-File system.
4- Maximum size of the database.
5-processes and threads and their responsibilities.
6-Query optimizer.
7-Cached execution path.
10-Stored procedures.
13-Table formats.
15-Data types.
17- The memory model I am going to use.

Of course I am not going to implement all these points (or even half of them!) , but as I said before, I want at least to put a design that can be modified to support them in the future.

Next time (God willing); I will talk about Security, so keep up with me.

Thursday, June 09, 2005

Let's practice

Recently, I have been reading some books and articles in the field of software engineering.
I have been reading Code Complete written by Steve McConnell and published by Microsoft press. This is an excellent book that can be classified as a book that touches software engineering. It mainly talks about the stage of code writing in the software process, but it tells you a lot of useful information about other stages too, such as requirements specification and design. In few words: This book is a must for software developers.
Also I have read the famous article No Silver Bullet by Fred Brook. This article is a landmark in the field of software engineering.
It explains why software industry is more difficult than other industries, and what solutions are proposed for such difficulties.
The Author, Fred Brook, is a software engineering professor at North Carolina computer science faculty, and he was one of the first people to work in the development of IBM operating systems. He used his experience in such great project to write his famous book Mythical Man Month which I am looking forward to read. In this book, Brook talks about aspects of software engineering of big projects generally, such as the addition of new members to the team and how this affect would the time and cost of the whole project.
A few days ago, I thought of a project to practice what I have learned and read recently. I thought of building a database engine - a small one, I am not going to compete with Oracle!
The reasons for this are:
1-Specifying requirements and design for such a big project is difficult.
2-I am looking forward to make a difficult system where the developer handles completely most things (memory, threads, processes, etc).
3-Although I have worked on two database systems (MySQL and SQL Server), my knowledge about the internals of database engines is very small. I have read some information about operating systems, but I have read nothing about database engines.

So the main goal I am expecting from such an experience is some practicing for what I have learnt and getting more knowledge in some fields of technology, I am not sure I will succeed in building a database engine in the end.

In the next blog, I am going to talk in detail about the project, specially the requirements specification of such a project. This may take a lot more than one blog to talk about!

Software : Dearest of all my friends!

Welcome to my technical blogs.
I am Mohammad Adel, an egyptian developer.I have been programming for 3 years now,I started on june 2002.I started development by learning C++, Direct x and OpenGL, because I was and still a hard-core gamer and I like any thing related to 3D computer graphics, either animated movies or games.
After that, I started learning .NET technology, C#, ASP.NET,XML.
Latelty I started learning software engineering and software security.
On the personal side, I am 21 years old.On My spare time, I play computer game, watch movies and play table tennis.
Regarding myBlogs, This is my first blog ever.My blogs are going to be only technical for those who are interested in software development. I am not going to speak about my personal life here - I am not Tom Cruise!
What I am going to speak about here may be in one of the following software fields or technologies:
3-Software security.
4-.NET Framework.
5-Software engineering.
6-Directx and 3D computer graphics.
Welcome to my blogs and waiting for your comments on them.