Database Source Control – How?

In this post I describe some decision that you need to make when putting your database development under source control.

“Commit a SQL script to a local repository and the push to a central repository” sounds easy. In fact the all process should be easy.sqlscript

The SQL script is the fundamental resource of the database source control process.

The script can represent two perspectives:

  • The desired definition of the database object (table, view, stored procedure, function…), a.k.a state-based approach;
  • The desired modification to be applied on the database object, a.k.a the migrations-based approach;

About this two approaches I like the definition presented by Alex Yates:

  • State: Your source of truth is how the database should be.
  • Migrations: Your source of truth is how the database should change.

So, this is your first big decision: state-based approach or migrations-based approach? Can I use both approaches? (The discussion of which one it’s the better approach will be explored on a future post).

The correct answer it’s directly related to… your budget. Each one of the solutions (state, migrations or even the combination of the two) will do the work and will cost time and money.

In your budget you should consider the following tools:

  • For state-based approach
    • Redgate SQL Source Control (+info)
    • SQL Server Data Tools (SSDT) (+info)
    • DBSourceTools (+info)
  • For migrations-based approach

The next decision is to choose the source control system. All the tools above identified support or can be used with the following source control systems:

  • Mercurial
    • Is a distributed source control management tool (+info)
  • Git
    • Distributed version control system designed to handle everything from small to very large projects with speed and efficiency (+info)
  • TFVC
    • Team Foundation Version Control (TFVC) is a centralized version control that lets you apply granular permission and restrict access down to a file level (+info)

As “normal” user, meaning that I do not use APIs to work with this systems, my preference was built through the use of GUIs:sourcecontrolsystems

  1. Mercurial ->  TortoiseHg (+info)
  2. Git -> SourceTree (+info)
  3. TFVC -> Visual Studio (+info)

 

Currently my database repositories are Mercurial or Git because I prefer distribute systems. Then, once I use the migrations-based approach my first choice is Mercurial/TortoiseHg because in my opinion have a better branch representation. It´s less powerful than Git making it easier to use and teach (I taught a telecommunications team use TortoiseHg with great success).

Summing up: to answer the question “How to put database under source control?” you need to make two main decisions:

  1. SQL scripting strategy
    • State-based or migrations-based approach
  2. Source Control System
    • Centralized or distributed system

 

Database Source Control – Why?

Nowadays the question “Why put database under source control?” no longer makes sense. Probably “How to put database under source control?” is now the correct question.

But before answering the second question let me answer the first identifying the reasons that led me/us team to invest on database source control:

  •   Is the first step of database automation
    • In other words, if you want to assemble a database deployment pipeline you should put your database changes into a source control system;
  • Keep history of each change
    • Allows traceability:
      • Everything went well: just do it and forget it;
      • Something was broken: you will need to know what changed, when was changed, why it was changed and who changed it;
  • Your best communication system
    • Allows to share code (SQL scripts);
    • In team context a source control system helps to manage possible conflict of interest;
  • Enforce standards (my reference was Sjors Takes)
    • Standards, preferably combined with automation, help to reduce/eliminate conflicts.

database

How to put database under source control? is the question that follows.

Database automation motivation

The first question I thought was: if applications development includes “by default” concepts as source control, continuous integration and continuous delivery, why not database development?

As Len Bass said in a conference: “A script is just a program”. All database changes can be scripted, so what is so special about databases? The answer is simple: data, specifically data persistence. One of my favorite answers was given by Grant Fritchey: “Deploying databases has the inherent problem of retaining the data after the deployment”.

Faced with this challenge the second question that I made was: why should I invest my time and effort in automating the database development process? I found the following  answers/reasons in literature and in my day to day job:

  •   Database are out of pace with application development
    • The database changes usually come from the development teams, however in most cases the DBA is responsible for applying this changes. This means that development teams and the DBA need to be synchronized, in the best case scenario this is a communication challenge;
  • No traceability of database changes (changes history)
    •  There will always be a moment where you need to answer the following questions: What was changed? Who changed it? When it was changed? Why it was changed?
    • In the ideal world the source control system should be able to answer this questions;
  • Manual database processes prevent the Continuous Integration (CI) and Continuous Delivery (CD) in their full extent
    • The strength of your process is equal to the strength of your weakest step of the process;
    •  Is acceptable to say that if the process contains a manual step, you can not say that you have a automated process? (Almost automated is not automated)
  • Manual database release process
    • This process is time consuming and error prone;
    • This make the release process less frequent and risky;
  • Tests are manual or do not exists
    • This have a serious impact in the application quality;
  • Bugs in production environment
    • Database related bugs are only discovered after deployment to production;
    • Fixes and hot-fixes have time cost, what can lead to delay a release;
  • Inability to roll back to the previous version of the database
  • Database setup in a new environment is time consuming
  • Databases become a bottleneck in agile delivery processes
    • Databases can be the most expensive component and the bigger troublemaker (part of it is because it’s “easy to blame” the database for the application problems).

The adoption of the database automation allows to:

  • Enable control over database development
    • If you have control, you have better conditions to improve;
  • Keep a versioned “history” of database states;
  • Increase release frequency through repeatability of processes
    •  Greater predictability over releases;
    • Efficiency by eliminating the repetition of a range of manual activities
  • Reduce time spent fixing bugs
    • With more control you are less expose to error;
  • Speed up response to change
    • Automated deployment of smaller units of change;
  • Remove/reduce human intervention in the release process
    • The build step is automatic triggered by a “push” into source control repository;
    • The deploy step is automatic triggered by a successfully build process;
  • Have greater reliability of the release process
    • Providing a series of automated test stages prior to deployment;
  • Maintain an agile development environment