.NET5_.NET Core, Uses database as configuration central server
This article introduces a way to use the database as the configuration central server on .NET, and describes a customized ConfigurationProvider, and explains the principle of its implementation.
1. Why use database as configuration center
When developing Youzack.com, a website for English learning, I needed to save configuration information such as AppKey of third-party interfaces, JWT etc. Youzack consists of login registration site, listening site, words recitation site, words recitation site(V2). To ensure the availability, the site uses cluster deployment, there are two Web server instances for one site, so the entire system deploys 2*4=8 instances. Configuration information is particularly cumbersome to manage if it is stored in a local configuration file. For example, if a configuration item needs to be changed, it needs to be changed in eight instances, so it needs to be saved to a configuration central server from which each application reads the configuration.
There are open source configuration centers such as Apollo, Nacos and Spring Cloud Config available, which are very feature-rich, but the configuration center server needs to be separately deployed and maintained. My site is not complicated, so I try to minimize the number of services I use in the site to avoid the hassle of operating them.
AliCloud also has a configuration center service that I can use, so I don’t have to deploy and maintain it myself, but I don’t want the site to be dependent on a specific cloud service provider, and that would require special treatment in the local development environment.
Considering that these subsites have to connect to the database, the configuration information stored in the database, with the database as the configuration center server, is in line with my requirements.
2. Advantages of my solution
Since the web site was developed in .NET 5, I developed a custom ConfigurationProvider, called Zack.AnyDbConfigProvider, to make it easier for each project to load the configuration.
The advantages of my solution are as follows.
1) The configurations are stored on the database, so it is easy to manage.
2) It supports all relational databases whichever .NET supports.
3) It supports version control of configurations;
4) It supports “the flattening of hierarchical data”.
5) The supported value types of configuration items are rich, including simple strings, numbers and other types, as well as JSON objects;
6) It’s developed in .Net standard2, so it can support.NET Framework,.NET Core, etc.
GitHub repository: https://github.com/yangzhongke/Zack.AnyDBConfigProvider
3. How to use Zack.AnyDBConfigProvider
Step one:
Create a table for retrieving configuration data from database. The table name is ‘t_configs’, which can be changed to other name with further configuration. The table must have there columns: Id(int, autoincrement), Name(text/varchar/nvarchar), Value(text/varchar/nvarchar).
Multi rows of the same ‘Name’ value is allowed, and the row with the maximum id value takes effect , which makes version controlling possible. The value of column ‘Name’ conform with “the flattening of hierarchical data”(https://docs.microsoft.com/en-us/aspnet/core/fundamentals/configuration/?view=aspnetcore-5.0), for example:
Api:Jwt:Audience
Age
Api:Names:0
Api:Names:1
The ‘Value’ column is for storing value of the corresponding value associated with that ‘Name’. The ‘Value’ can be plain value, string value of json array, and string value of json object. for example:
[“a”,”d”]
{“Secret”: “afd3”,”Issuer”: “youzack”,”Ids”:[3,5,8]}
ffff
3
This is the data that will be used in the following demonstrations:
Step Two
Please create an ASP.NET project. The demo project here is created by Visual Studio 2019 in .NET Core 3.1. However, Zack.AnyDBConfigProvider is not limited to this version.
Install the package via NuGet:
Install-Package Zack.AnyDBConfigProvider
Step Three: Configure the connection string of the database
While the other configurations in the project can be stored in the database, the connection strings for the database itself still need to be configured separately. It can be configured either in a local configuration file, or through environment variables. Let’s take using a local JSON file for example.
Edit appsettings.json, and add the following codes:
“ConnectionStrings”: {
“conn1”: “Server=127.0.0.1;database=youzack;uid=root;pwd=123456”
},
Then, insert the following code before webBuilder.UseStartup<Startup>(), which is located in CreateHostBuilder of Program.cs:
webBuilder.ConfigureAppConfiguration((hostCtx, configBuilder)=>{
var configRoot = configBuilder.Build();
string connStr = configRoot.GetConnectionString(“conn1”);
configBuilder.AddDbConfiguration(() => new MySqlConnection(connStr),reloadOnChange:true,reloadInterval:TimeSpan.FromSeconds(2));
});
The third line of the above code reads the connection string for the database from the local configuration, and the fourth line uses AddDbConfiguration()to add support for Zack.AnyDBConfigProvider. I’m using the MySQL database, so I creates a connection to the MySQL database using “new MySqlConnection(connStr)”. You can change it to any other database management system you want. The reloadOnChange parameter indicates whether the configuration changes in the database are automatically reloaded. The default value is false. If reloadOnChange is set to true, then every reloadInterval, the program will scan through the database configuration table, if the database configuration data changes, it will reload the configuration data. The AddDbConfiguration() method also supports a tableName parameter as the name of the self-defined configuration table, the default name is T_Configs.
Different versions of the development tools generate different project templates, so the initial code might be different, so the above code may not be put in your project intact, please customize the configuration code according to your own project.
Step four:
Then we can use the standard approach of .NET to read configurations. For example, if we want to read the data in the above example, we would configure it as follows.
First, please create a FTP class (with properties of IP, Username, Password), and a Cors class (with properties of string[] Origins and string[] Headers).
Then, add the following code into ConfigureServices() of Startup.cs.
services.Configure<Ftp>(Configuration.GetSection(“Ftp”));
services.Configure<Cors>(Configuration.GetSection(“Cors”));
And, use the following code to read configurations:
public class HomeController : Controller
{
private readonly ILogger<HomeController> _logger;
private readonly IConfiguration config;
private readonly IOptionsSnapshot<Ftp> ftpOpt;
private readonly IOptionsSnapshot<Cors> corsOpt;
public HomeController(ILogger<HomeController> logger, IConfiguration config, IOptionsSnapshot<Ftp> ftpOpt, IOptionsSnapshot<Cors> corsOpt)
{
_logger = logger;
this.config = config;
this.ftpOpt = ftpOpt;
this.corsOpt = corsOpt;
}
public IActionResult Index()
{
string redisCS = config.GetSection(“RedisConnStr”).Get<string>();
ViewBag.s = redisCS;
ViewBag.ftp = ftpOpt.Value;
ViewBag.cors = corsOpt.Value;
return View();
}
}
I won’t cover how to use the read configuration. I will display the configuration values on the screen only. You can modify the configuration, and then refresh the interface, you can see the modified values on the screen.
4. What’s under the hood?
The github repository is https://github.com/yangzhongke/Zack.AnyDBConfigProvider, and the core class of this project is DBConfigurationProvider.
All custom configuration providers in .NET should implement the IConfigurationProvider interface, and they usually derive directly from the abstract class ConfigurationProvider. The most important method of ConfigurationProvider is the Load() method, which the custom ConfigurationProvider implements to load data. The Load() method will store all configurations into the Data property as a key-value pair. The type of Data property is IDictionary<string, string>, and the Key is the name of the configuration, following the “the flattening of hierarchical data” specification. The OnReload () method is called to notify the listeners when configurations are changed.
The basic mechanism of the ConfigurationProvider class has been described above. Let’s examine the main code of the Zack.AnyDBConfigProvider.
First, let’s check out the constructor of DBConfigurationProvider:
ThreadPool.QueueUserWorkItem(obj => {
while (!isDisposed)
{
Load();
Thread.Sleep(interval);
}
});
As you can see, if reloadOnChange is enabled, Load() is called to reload the data every specified time.
The main code of the Load() method is as follows:
public override void Load()
{
base.Load();
var clonedData = Data.Clone();
string tableName = options.TableName;
try
{
lockObj.EnterWriteLock();
Data.Clear();
using (var conn = options.CreateDbConnection())
{
conn.Open();
DoLoad(tableName, conn);
}
}
catch(DbException)
{
//if DbException is thrown, restore to the original data.
this.Data = clonedData;
throw;
}
finally
{
lockObj.ExitWriteLock();
}
//OnReload cannot be between EnterWriteLock and ExitWriteLock, or “A read lock may not be acquired with the write lock held in this mode” will be thrown.
if (Helper.IsChanged(clonedData, Data))
{
OnReload();
}
}
The main idea of the Load() method is to create a copy of the Data attribute first, which can be used to compare “whether the Data has changed” later. Because if reloadOnChange is enabled, Load() method will be called periodically in a thread, and the code that reads the configuration will eventually call the TryGet() method to read the configuration. To prevent TryGet from reading the data that are been processed by Load() method, locks are needed to control the synchronization of reads and writes. Since reads are usually more frequent than writes, to avoid performance problems with normal locks, the ReaderWriterLockSlim class is used to implement “only allow one thread to write, but allow multiple threads to read”. I put the code that loads the configuration to write to the Data property between EnterWriteLock() and ExitWriteLock(), and wrap the code that reads the configuration (see the TryGet method) with EnterReadLock() and ExitReadLock().
It’s worth noting that in the Load() method, be sure to put OnReload () after ExitWriteLock(), otherwise it will cause the “A read lock may not be acquired with the write lock held in this mode” exception. Because the OnReload method will cause the invocation of TryGet() to read the data, and TryGet() method will use a “read lock,” this results in a “read lock” nested in a “write lock,” which is not allowed by default.
In the DoLoad() method, data is read from the database and loaded into the Data property. At the end of the Load() method, the saved copy of the Data attribute clonedData is compared with the new value of Data property. If the Data has changed, OnReload () will be called, “The Data has changed, please load the new Data.”
The DoLoad() method is to load the configured value into the Data attribute. Although there are many codes, the codes are not complicated. It is mainly to parse and load the Data according to the specification of “the flattening of hierarchical data”. Because I did not fully understand this specification before, I took some detours. This is my one of the highlights of this open source project, because if it is only in accordance with the“the flattening of hierarchical data” specification strictly, the database name must be “Ftp: IP”, “Ftp: UserName”, “Cors: Origins: 0” and “Cors: Origins: 1”, “Cors: Origins: 2”, but after my processing, configuration values can be used in very readable json formats (of course, it’s still compatible with strict “the flattening of hierarchical data” specification).
5. Conclusion
Zack.AnyDBConfigProvider is an open-source library that uses the database as configuration center server. It allows you to configure projects in a simple version-managed configuration center in a highly readable format without adding additional configuration center servers. I hope this open-source project can help you. Please feel free to give feedback. If you like it, please don’t hesitate to recommend it to your friends.
