A Tale of Two Repos
Two repos, one goal !
When I was a child, my mother used to tell nice bedtime stories. My favorite one was the The Tale of Two Repos. It goes like this :
"Once upon a time, in the CdeP, a little
core.dashboards_storerepo was born. The repo tried very hard to provide ETLs for all the school boards at once. But the school boards were many and the SQL capabilities to retrofit itself to a school board's context were few. So thecore.dashboards_storerepo decided to split itself into many little repos, one for each school board. And thecssXX.dashboards_storerepo was born. And thecssXX.dashboards_storerepo was granted the power to override anything fromcore.dashboards_store. And thecore.dashboards_storerepo was happy. And thecssXX.dashboards_storerepo was happy. And the CdeP was happy. And they all merged develop into master happily ever after."
The moral of the story is that the core.dashboards_store repo is the parent repo of all of the cssXX.dashboards_store repos. The core.dashboards_store repo contains all the ETLs that are common to all of the CSS. The cssXX.dashboards_store repo contains everything that is specific to cssXX, including, if needed, some SQL files overriding the ones in the core.
If you have already been exposed to object-oriented programming, the core.dashboards_store repo is the parent class providing overridable methods, and the cssXX.dashboards_store repo is the child class that might want to override some of the parent's methods.
core.dashboards_store: the core.dashboards_store repo containing everything that is shared between school boards, and your own custom cssXX.dashboards_store repo implementing what can't be shared, such as the way to identify students enrolled in an adapted program.cssXX.dashboards_store repo. The core.dashboards_store repo should never be modified as it is shared between all school boards.Cloning the repo
<working directory> of your choice. If not, you will have to change the cssXX.dashboards_store/packages.yaml to reflect the new path.Bootstrapping a working Dashboards Store
- Start by cloning the Core: (You'll need a properly set up GitHub account with an SSH key).
git clone git@github.com:Sciance-Inc/core.dashboards_store.git
- Check out the master branch to get the latest template revision
git checkout master
- Create a
Poetry / Virtualenvfrom the Core. This will install every Python package you need, including dbt.
cd core.dashboards_store
eval $(poetry env activate)
poetry install --no-root # The --no-root might not be needed, depending on your Poetry version [#38](https://github.com/Sciance-Inc/core.dashboards_store/issues/38)
- Bootstrap a new
cssXX.dashboards_storerepo from the Core's templateYou will be prompted for information
cd ../
cookiecutter core.dashboards_store/tooling/template
- Let's follow best practices and use Git for the repo you have just created
git init
git remote add origin <your remote's url>
git add .
git commit -m "feat: one commit to initiate them all, one commit to rule them all, one commit to bring them all and in the gitness bind them, in the land of GitHub where the bugs lie."
git push -u origin master
- Read the
cssXX.dashboards_store/README.mdto learn more about the post-configuration steps required to have everything working.
What did I just do ?
If everything worked well, you should have the following directory structure:
<working directory>
├── core.dashboards_store
└── cssXX.dashboards_store
Both subfolders should contain Git-ready dbt projects.
<working directory> will always designate the path to the folder containing the two repos (unless mentioned otherwise).Machine set-up
Database connection
For now, we haven't connected the Store to the database. To connect the Store to the database and start materializing our views and tables, we need to tell dbt where to look for the database credentials and connection information. We do this by creating a profiles.yml file.
