Posted onJul 5, 2022 • Edited onSep 1, 2022

File Database in Node Js from scratch part 2: Select function & more

#javascript #node #webdev #programming

introduction

Welcome to part 2 of trying to build from scratch and becoming a better programmer, if you just stumbled upon this post and have no idea what's going you can find part 1here, else welcome back and thank you for your time again.

Part 1 was just a setup, nothing interesting happened really and since then I had some time to think about stuff, hence some refactor and lot of code in this part.

Database.js

Soon to be previous db function:

functiondb(options){this.meta={length:0,types:{},options}this.store={}}

The first problem I noticed here isthis.store is referenced a lot in operations.js by different functions, initially it may not seem like a big deal, but if you think for a minute as object are values by reference, meaning allowing access to a single object by multiple functions can cause a huge problem, like receiving an outdated object, trying to access a deleted value etc,

The functions(select, insert, delete_, update) themselves need to do heavy lifting making sure they are receiving the correct state, checking for correct values and so on, this leads to code duplication and spaghetti code.

I came up with a solution inspired by state managers, having a single store which exposes it's own API, and no function outside can access it without the API.

The API is responsible for updating the state, returning the state and maintaining the state, any function outside can request the store to do something and wait, code speaks louder than words, here is a refactored db function

importStorefrom"./Store.js"functiondb(options){this.store=newStore("Test db",options)// single endpoint to the database}

I guess the lesson here is once everything starts getting out of hand and spiraling, going back to the vortex abstraction and creating a single endpoint to consolidate everything can be a solution. This will be apparent once we work on the select function.

one last thing we need is to remove select from operators to it's own file, select has a lot of code

updated Database.js

import{insert,update,delete_}from'./operators.js'// remove selectimportStorefrom"./Store.js"importselectfrom"./select.js"// new selectfunctiondb(options){// minor problem: store can be accessed from db object in index.js// not a problem thou cause #data is privatethis.store=newStore("Test db",options)}db.prototype.insert=insertdb.prototype.update=updatedb.prototype.select=selectdb.prototype.delete_=delete_exportdefaultdb

Store.js (new file)

I chose to use a class for store, you can definitely use a function, my reason for a class is, it is intuitive and visually simple for me to traverse, and easy to declare private variables

If you are unfamiliar with OOJS(Object Oriented JS) I do have two short articleshere, and for this article you need to be familiar with thethis keyword

exportdefaultclassStore{// private variables start with a "#"#data={}#meta={length:0,}// runs immediatley on class Instantiationconstructor(name,options){this.#meta.name=name;this.#meta.options=options}// API// getters and setters(covered in OOJS)//simply returns datagetgetData(){returnthis.#data}// sets data// data is type ObjectsetsetData(data){data._id=this.#meta.lengthif(this.#meta.options&&this.#meta.options.timeStamp&&this.#meta.options.timeStamp){data.timeStamp=Date.now()}this.#data[this.#meta.length]=datathis.#meta.length++}}

Explaining setData

data._id=this.#meta.length// _id is reserved so the documents(rows) can actually know their id's

adding timeStamp

if(this.#meta.options&&this.#meta.options.timeStamp&&this.#meta.options.timeStamp){data.timeStamp=Date.now()}// this lines reads// if meta has the options object// and the options object has timeStamp// and timeStamp(which is a boolean) is true// add datetime to the data before commiting to the db// this check is necessary to avoid cannot convert null Object thing error

// putting the document or rowthis.#data[this.#meta.length]=data// incrementing the id(pointer) for the next rowthis.#meta.length++

Now we can safely say we have a single endpoint to the database(#data) outside access should consult the API, and not worry about how it gets or sets the data

However using setData and getData sounds weird, we can wrap these in familiar functions and not access them directly

classes also have a proto object coveredhere

Store.prototype.insert=function(data){// invoking the setter// this keyword points to the class(instantiated object)this.setData=data}

now we can update operators.js insert

operators.js

// updating insert(letting the store handle everything)exportfunctioninsert(row){this.store.insert(row)}

Select.js

I had many ideas for select, mostly inspired by other db's, but I settled on a simple and I believe powerful enough API, for now I want select to just do two things select by ID and query the db based on certain filters.

let's start with select by id as it is simple

exportdefaultfunctionselect(option="*"){// checking if option is a numberif(Number(option)!==NaN){// return prevents select from running code below this if statement()// think of it as an early breakreturnthis.store.getByid(+option)// the +option converts option to a number just to make sure it is}// query mode code will be here}

based on the value of option we select to do one of two select by ID or enter what i call a query mode, to select by id all we need to check is if option is a number, if not we enter query mode

Store.js

we need to add the select by id function to the store

...Store.prototype.getByid=function(id){constdata=this.getData// get the pointer the data(cause it's private we cannot access it directly)//object(remember the value by reference concept)if(data[id]){// checking if id existsreturndata[id]// returning the document}else{return"noDoc"// for now a str will do// but an error object is more appropriate(future worry)}}

Simple and now we can get a row by id, query mode is a little bit involved, more code and some helpers

Select.js query mode

The core idea is simple really, I thought of the db as a huge hub, a central node of sort, and a query is a small node/channel connected to the center, such that each query node is self contained, meaning it contains it's own state until it is closed.

example

leta=store.select()// returns a query chanel/nodeletb=store.select()// a is not aware of b, vice versa,//whatever happens in each node the other is not concerned

for this to work we need to track open channels and their state as the querying continues, an object is a simple way to do just that.

consttracker={id:0,// needed to ID each channel and retrieve or update it's state}functionfunctionalObj(store){this.id=NaN// to give to tracker.id(self identity)}exportdefaultfunctionselect(option="*"){...// query mode code will be here// functionalObj will return the node/channelreturnnewfunctionalObj(this.store)}

functionalObj will have four functions:

beginQuery - will perform the necessary setup to open an independent channel/node to the db

Where - will take a string(boolean operators) to query the db e.gWhere('age > 23') return all docs where the age is bigger than 23

endQuery - returns the queried data

close - destroys the channel completely with all it's data

beginQuery

...functionfunctionalObj(store){...// channelName will help with Identifying and dubugging for the developer using our dbthis.beginQuery=(channelName="")=>{// safeguard not to open the same query/channel twiceif(tracker[this.id]&&tracker[this.id].beganQ){// checking if the channel already exists(when this.id !== NaN)console.warn('please close the previous query');return}// opening a node/channelthis.id=tracker.idtracker[this.id]={filtered:[],// holds filtered databeganQ:false,// initial status of the channel(began Query)cName:channelName===""?this.id:channelName}tracker.id++// for new channels// officially opening the channel to be queried// we will define the getAll func later// it basically does what it's saystracker[this.id].filtered=Object.values(store.getAll())// to be filtered datatracker[this.id].beganQ=true// opening the channelconsole.log('opening channel:',tracker[this.id].cName)// for debugging}// end of begin query function}

update Store.js and put this getAll func

Store.prototype.getAll=function(){returnthis.getData}

Where, endQuery, close

functionfunctionalObj(store){this.beginQuery=(channelName="")=>{...}// end of beginQuerythis.Where=(str)=>{// do not allow a query of the channel/node if not openedif(!tracker[this.id]||tracker[this.id]&&!tracker[this.id].beganQ){console.log('begin query to filter')return}letf=search(str,tracker[this.id].filtered)// we will define search later(will return filtered data and can handle query strings)// update filtered data for the correct channelif(f.length>0){tracker[this.id].filtered=f}}// end of wherethis.endQuery=()=>{if(!tracker[this.id]||tracker[this.id]&&!tracker[this.id].beganQ){console.warn('no query to close')return}// returns datareturn{data:tracker[this.id].filtered,channel:tracker[this.id].cName}};// end of endQuerythis.close=()=>{// if a node/channel exist destroy itif(tracker[this.id]&&!tracker[this.id].closed){Reflect.deleteProperty(tracker,this.id)// deleteconsole.log('cleaned up',tracker)}}}

Search

// comm - stands for commnads e.g "age > 23"constsearch=function(comm,data){letsplit=comm.split("")// ['age', '>', 23]// split[0] property to query// split[1] operator// compare againstletfiltered=[]// detecting the operatorif(split[1]==="==="||split[1]==="=="){data.map((obj,i)=>{// mapSearch maps every operator to a function that can handle it// and evalute it// mapSearch returns a boolean saying whether the object fits the query if true we add the object to the filteredif(mapSearch('eq',obj[split[0]],split[2])){// e.g here mapSearch will map each object with a function// that checks for equality(eq)filtered.push(obj)}})}elseif(split[1]==="<"){data.map((obj,i)=>{// less than searchif(mapSearch('ls',obj[split[0]],split[2])){filtered.push(obj)}})}elseif(split[1]===">"){data.map((obj,i)=>{// greater than searchif(mapSearch('gt',obj[split[0]],split[2])){filtered.push(obj)}})}returnfiltered// assigned to f in Where function}functionfunctionalObj(store){...}

mapSearch

// direct can be eq, gt, ls which directs the comparison// a is the property  --- age// b to compare against --- 23constmapSearch=function(direct,a,b){if(direct==="eq"){// comparers defined func belowreturncomparers['eq'](a,b)// compare for equality}elseif(direct==="gt"){returncomparers['gt'](a,b)// is a > b}elseif(direct==="ls"){returncomparers['ls'](a,b)// is a < b}else{console.log('Not handled')}}constsearch=function(comm,data){...}...

Comparers

actually does the comparison and returns appropriate booleans to filter the data

// return a boolean (true || false)constcomparers={"eq":(a,b)=>a===b,"gt":(a,b)=>a>b,"ls":(a,b)=>a<b}

Select should work now, we can query for data through dedicated channels

test.js

testing everything

importdbfrom'./index.js'letstore=newdb({timeStamp:true})store.insert({name:"sk",surname:"mhlungu",age:23})store.insert({name:"np",surname:"mhlungu",age:19})store.insert({name:"jane",surname:"doe",age:0})constc=store.select()// return a new node/channel to be openedc.beginQuery("THIS IS CHANNEL C")// opening the channel and naming itc.Where('age < 23')// return all documents where age is smaller than 23constd=store.select()// return a new node/channeld.beginQuery("THIS IS CHANNEL D")// open the channeld.Where('age > 10')// all documents where age > 10console.log('===============================================')console.log(d.endQuery(),'D RESULT age > 10')// return d's dataconsole.log('===============================================')console.log(c.endQuery(),"C RESULT age <  23")// return c's dataconsole.log('===============================================')c.close()// destroy cd.close()// destroy d

node test.js

you can actually chain multiple where's on each node, where for now takes a single command

example

constc=store.select()c.beginQuery("THIS IS CHANNEL C")c.Where("age > 23")c.Where("surname === doe")// will further filter the above returned documents

Problems

the equality sign does not work as expected when comparing numbers, caused by the number being a string

// "age === 23"comm.split("")// ['age', '===', '23'] // 23 becomes a string23==='23'// returns false// while 'name === sk' will workcomm.split("")// ['name', '===', 'sk']'sk'==='sk'

a simple solution will be to check if each command is comparing strings or numbers, which in my opinion is very hideous and not fun to code really, so a solution I came up with is to introduce types for the db, meaning our db will be type safe, and we can infer from those types the type of operation/comparisons

for example a new db will be created like this:

letstore=newdb({timeStamp:true,types:[db.String,db.String,db.Number]// repres columns})// if you try to put a number on column 1 an error occurs, because insert expect a string

the next tut will focus on just that.

conclusion

If you want a programming buddy I will be happy to connect ontwitter , or you or you know someone who is hiring for a front-end(react or ionic) developer or just a JS developer(modules, scripting etc)I am looking for a job or gig please contact me:mhlungusk@gmail.com, twitter will also do

Thank you for your time, enjoy your day or night. until next time