Menu

data engineering

Explore our insightful blog posts that delve deeper into the latest industry trends, best practices, and success stories. Gain valuable knowledge and stay informed about the ever-evolving landscape of digital transformation.

Suggested categories

Mastering Data Analysis Skills and it’s Impact on Business

Are you curious about what it takes to become a data analyst? Being a data analyst is an increasingly important role in today’s business landscape and the ability to develop insights from data can be invaluable for companies. In this blog, we’ll discuss what a data analyst does. We will also explore some of the data analysis skills and technologies that would help you become one. We’ll also touch on how even project-based learning can give you an edge as an aspiring data analyst.Who is a data analyst?A data analyst is someone who collects and analyzes data from various sources to uncover patterns, trends, and insights. They are responsible for understanding a company’s internal or external data sets, extracting meaningful information from them, and presenting it in an easy-to-understand way. Data analysts use analytical tools such as SQL and statistical software like SPSS to help them draw conclusions about the data they analyze. By doing this work, they can help businesses make better-informed decisions that will drive their success.What technologies does a data analyst need to learn?Data analysts need to be well-versed in many different technologies and tools. They should have a good understanding of SQL, which allows them to query databases and extract data. Additionally, they will need to learn statistical software like SPSS or SAS that can help them analyze the data they pull out of databases. Data analysts also need to understand how to use visualization tools such as Tableau and Power BI, so they can present their findings in an easy-to-understand way. Finally, data analysts may benefit from learning programming languages such as Python or R, which allow them to automate repetitive tasks or build more complex models with their data sets.Importance of a data analyst in the current business landscapeData analysts are playing an increasingly important role in today’s business climate. As companies become more data-driven, organizations need people who can extract meaningful insights from their data sets. Data analysts can help businesses make better-informed decisions by providing them with a deeper analysis of their internal or external data sources. They provide a valuable link between the raw data and decision-makers within a company, giving them the information they need to move forward confidently with any kind of strategic plan.They also play a pivotal role in  consuming and processing large amounts of data generated by the Engineering or Platforms team and then synthesizing it to feed the Data Science Team which powers Digital Transformation initiatives in the organization. In addition to this, data analysts can also identify potential areas for improvement that could lead to increased profits or efficiency gains for the business. Ultimately, having a skilled and experienced data analyst on board is essential for any organization looking to stay ahead of its competition in today’s rapidly evolving landscape.How can project-based learning help you become a data analyst?Project-based learning is an excellent way to gain experience and refine your skills as a data analyst. Through project-based learning, you can get hands-on practice with the tools and technologies used in data analysis. You can work on real-world projects that require you to solve complex problems or build models from scratch, allowing you to apply what you’ve learned in a practical setting. Additionally, project-based learning provides a great platform for networking with other professionals in the field and getting valuable feedback on your progress. By working through projects related to data analysis, you’ll be able to hone your technical abilities while gaining insight into the industry as well. Project-based learning could be just the thing needed to set yourself apart from other aspiring data analysts!Overall, a data analyst is an important role in the current business world. They need to be equipped with numerous technologies and tools to collect, analyze, archive and visualize large datasets. Project-based learning is one of the best pathways to becoming a proficient data analyst as it will allow you to practice the different techniques first-hand to apply them effectively in real-world scenarios. With the right practice and dedication, you can acquire all the skills necessary for making a successful career out of this job!Are you an aspiring data analyst looking to start your career in a dynamic data-driven work environment? Get in touch.

Learn More >

Understanding the Data Engineering Evolution

The past decade has exponentially changed the way that data is perceived, and used. Organizations are pursuing and implementing data-centric initiatives and are relying more and more on data-driven decisions to create business value and innovation. The focus of this blog will be the Data Engineering evolution from managing data inflow/outflow from databases to making it more accessible and providing the right data to its users at the right time.The EvolutionData engineering was brought into the main picture a few years ago when the increased demand for data handling and engineering support emerged with the shifting data landscape. However, the original conceptualization of the discipline can be traced back to the late ’90s. At the time, data engineering was a subset of a number of emerging data technologies and techniques in the analytics landscape. It dealt with the use of ETL (Extract, Transform and Load) and managed the mobility of data to various channels such as databases and warehouses.It was in the early 2000s that the requirement of scalability made itself pronounced upon the data landscape as the easy accessibility of the internet brought an increased online engagement between companies and their consumers. This ushered in a new age for data as opportunities for analytics increased dramatically. Organizations began focusing on collecting and storing data through the use of data lakes. Newer technologies such as Apache Hive gained popularity and gave way to increased flexibility and scalability.In 2006, it became easier and cheaper to store and manage huge amounts of data with the arrival of Big Data and its sibling technologies. Hadoop open-sourcing became a turning point in the way data was being processed. The new complexities behind processing the data gave birth to a new breed of data and backend engineers. Hadoop played an important part in building the storage layer S3 of Amazon Web Services when it was launched in 2006. The second turning point that bridged the gap between these engineers and Big data was when Hive open-sourced in 2010 and this led to a new era of Data engineering.This new era brought forth complexities in data sourcing from different storage locations and organizations were subjected to a new challenge of operating this complex flow of data. This challenge gave rise to data orchestration engines that allowed the flow of data from multiple data storage locations and combined it to make it easily available for various data operations.This massive explosion of data helped usher in a new age for machine learning. Machine learning models transitioned from being trained on a single machine to being trained on the abundance of data collected from the internet. This further evolved in 2014 with the release of MLlib by Spark for Python which led to the democratization of ML computation on Big Data. In addition to this, Spark offered a new direction for data engineers to compute and process streaming data with relative ease and thus advancing towards the era of real-time processing. 2014 was a significant year in the development of how data can be used. In addition to Spark’s impact, the introduction of the Lambda function on Amazon Web Services gave rise to the serverless movement where data ingestion could be easily done without infrastructure management. This allowed data engineers to take a break from managing infrastructure and spend considerably more time on scaling and development.  In 2016 the new release, Athena helped propagate things further by allowing to query directly onto s3 without the need to set up a cluster.The data landscape today has been subjected to a number of iterations and this has changed the way data is now sourced, collected, and processed. In addition to working with data from external sources, organizations of today have to worry about data that is generated internally. The complexity has substantially increased to make sure that the right data is available for experimentation. A Gartner Data Science Team Survey 2018 shows that for data projects, a substantial amount of time goes into tasks such as data collection and preparation, problem analysis, before commencing the development of the various data models. Data engineering has become a critical practice that helps bridge the gaps around the accessibility of data and to ensure success in data and analytics initiatives.With the gap between newer innovations growing shorter and smaller, the existing layers of data and technology keep shifting towards adaptability. The foreseeable increase in the data volumes from various data sources and collection points will pave the way for newer techniques in reconfiguring data into usable forms. It is becoming vital for organizations to embrace data engineering practice to drive data and analytics success.

Learn More >

Mastering Clojure Fundamentals for Beginners

1. Getting Started1.1. Running the Clojure InterpreterClojure is powerful, easy to use, general purpose programming language that runs on the JVM. Clojure like other Lisp dialects is dynamic, which means that it is not overly typed based system and gives you some room to breathe. In addition to that Clojure is compiled and can easily call upon the vast amount of Java libraries and the goodness of the JVM. I have created this series keeping in mind the average programmer who is more experienced in C-based programming languages like Java, Python, JavaScript. If you come from such programming languages and are looking into exploring the world of a full-fledged functional programming language, then Clojure is the one for you.This chapter is the starting of a series on Clojure. We will try to see the internals of Clojure and at the end of it seek to build a real world app.You can run Clojure both as a jar file or as an interpreter. We will look at how to distribute the Clojure code as a jar file later on. Right now lets first start Clojure as an interpreter. To do that install leiningen which is considered to be the easiest way to use Clojure.First, you need to install Java if you don’t have that installed already. The below instructions are for an Ubuntu machine. In case you have a different machine and are not able to install the dependencies, write to me and we will try to find what the issue might be. sudo apt-get update sudo add-apt-repository ppa:webupd8team/java sudo apt-get update sudo apt install oracle-java8-installerview rawinstall_java8_ubuntu.bash hosted with ❤ by GitHubYou can then follow the installation steps as shown below. This is the same as the official installation steps which I followed in my Ubuntu machine. ➜ ~ wget https://raw.githubusercontent.com/technomancy/leiningen/stable/bin/lein --2017-06-14 19:57:44-- https://raw.githubusercontent.com/technomancy/leiningen/stable/bin/lein Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.8.133 Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.8.133|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 12871 (13K) [text/plain] Saving to: ‘lein’   lein 100%[==========================================================>] 12.57K --.-KB/s in 0.007s   2017-06-14 19:57:45 (1.72 MB/s) - ‘lein’ saved [12871/12871]   ➜ ~ sudo mv lein /usr/local/bin/lein [sudo] password for joy: ➜ ~ sudo chmod a+x /usr/local/bin/leinview rawinstall_leiningen.bash hosted with ❤ by GitHubOnce done you can start the lein repl command and this will download the necessary libraries and start a repl for you. ➜ ~ lein repl nREPL server started on port 37057 on host 127.0.0.1 - nrepl://127.0.0.1:37057 REPL-y 0.3.7, nREPL 0.2.12 Clojure 1.8.0 Java HotSpot(TM) 64-Bit Server VM 1.8.0_131-b11 Docs: (doc function-name-here) (find-doc "part-of-name-here") Source: (source function-name-here) Javadoc: (javadoc java-object-or-class-here) Exit: Control+D or (exit) or (quit) Results: Stored in vars *1, *2, *3, an exception in *e   user=>view rawlein_repl.bash hosted with ❤ by GitHubNow check if you can run the code and if it gives the correct result.user=> (+ 1 1)2Voila! You have successfully started your Clojure interpreter and have run a Clojure program.1.2 AssignmentsThe use of global variables is recommended to avoid in Clojure and to keep the functions as clean and pure as possible. But you can create a persistent reference to a changing object. This is done through the use of Vars. user=> (def x 1) #'user/x user=> x 1 user=> (def y 2) #'user/y user=> (+ x y) 3 user=>view rawasignment.clj hosted with ❤ by GitHubVars are generally static. So, if you need to make it dynamic and have some calculations based on them, you will need to use the macro binding. Bindings are thread-local and do not propagate outside the thread. user=> (def ^:dynamic x 1) #'user/x user=> (def ^:dynamic y 1) #'user/y user=> (binding [x 2 y 3] (+ x y)) 5 user=> (+ x y) 2 user=>view rawdunamic_vars.clj hosted with ❤ by GitHub1.3 NumbersClojure is a dialect of Lisp and that means that means that operators are considered as functions and so come before the arguments that should go into the operation. user=> (+ 1 1) 2 user=> (* 2 3) 6 user=> (/ 17 5) 17/5 user=> ( float (/ 17 5)) 3.4view rawnumbers.clj hosted with ❤ by GitHub1.4 StringsStrings are believed to be one of the basic building blocks of programming. Almost everything that you do will involve manipulation of strings.In Clojure, a string literal is created by enclosing the string in double quotes.To concatenate some string variables you can use the `str` function: user=> (def a1 "My favorite lang is ") #'user/a1 user=> (def a2 "Clojure") #'user/a2 user=> (str a1 a2) "My favorite lang is Clojure" user=>view rawstrings.clj hosted with ❤ by GitHubYou can also use string formatting using the Format function. user=> (format "Any man whose errors take %s years to correct is quite the man." "10") "Any man whose errors take 10 years to correct is quite the man." user=>view rawstring_format.clj hosted with ❤ by GitHub1.5 FunctionsClojure is a functional programming language. So you will be creating a lot of functions. Functions that do something, functions that pass functions, functions that manipulate other functions and so on. You can consider a function to be unrealized functionality. Generally, functions are defined in the following manner.(defn name doc-string? attr-map? [params*] body)For example, a function that returns a square of a number can be created. (defn my_square "This is my my_square function" [x] (* x x))   (my_square 2)view rawsquare.clj hosted with ❤ by GitHubThe above code should return 4. Now the functions can be used in other expressions.(+ (my_square 2) (my_square 3))which should give the output 13.If you just need to create a small function that you can consider writing anonymous functions usingfn .user=> ((fn [x] (* x x)) 2)41.6 ConditionsOne way of implementing branching in Clojure is through the if statement which is basically a test. The format is basically:(if condition-statment <(optional)if condition is false return this>)The above construct will be clear in the below examples.user=> (if nil “truth value” “false value”)“false value”In Clojure, only nil and false are considered logical false.user=> (if “true” “truth value” “false value”)“truth value”But if cannot take multiple tests in the format of if..elif..else. For that you will need to use thecond macro. Then you can pass multiple test parameters. user=> (defn odd-even #_=> "Determines whether or not n is odd or even." #_=> [n] #_=> (cond #_=> (= (rem n 2) 0) "even" #_=> (= (rem n 2) 1) "odd")) #'user/odd-even user=>   user=> (odd-even 2) "even" user=> (odd-even 3) "odd"view rawmultiple_conditions.clj hosted with ❤ by GitHubThe above constructs are more in line with the imperative languages.Of course, you can convert any if constructs to filter, but more on that later.1.7 Data Containers and Data Structures.Clojure has four primary data structure or collections of data.1.Lists2.Vectors3.Maps4.SetsLists in Clojure are singly linked lists. So modifications of lists are efficient while random access is not. Lists are one of the most important data structures in Clojure. A code is data in Clojure and they are represented in lists.You can either use the list function or the quote literal to creating a list.user=> (list “clojure” “F#” “python” “java”)(“clojure” “F#” “python” “java”)user=> ‘(“clojure” “F#” “python” “java”)(“clojure” “F#” “python” “java”)Get the first or nth element of the list. user=> (def langs (list "clojure" "F#" "python" "java")) #'user/langs user=> (first langs) "clojure" user=> (nth langs 2) "python"view rawlist_lookup.clj hosted with ❤ by GitHubYou can treat the list like a stack with peek and pop. user=> (peek langs) "clojure" user=> langs ("clojure" "F#" "python" "java") user=> (pop langs) ("F#" "python" "java") user=>view rawas_stack.clj hosted with ❤ by GitHubYou can also extend a list to a new list with conj and concat. Notice that the conjoining in lists happen at the beginning of the list. user=> (conj langs "APL") ("APL" "clojure" "F#" "python" "java") user=> (conj langs '("OCAML" "erlang")) (("OCAML" "erlang") "clojure" "F#" "python" "java") user=> (concat langs '("OCAML" "erlang")) ("clojure" "F#" "python" "java" "OCAML" "erlang")view rawlist_manipulation.clj hosted with ❤ by GitHubB. VectorsVectors are similar to arrays in C and Java. The elements in a vector are indexed by contiguous integers. Vectors are optimized for random lookups.To create a vector you can use the vector function or just define them. user=> (vector "Amherstia Nobilis" "Adansonia digitata" "Ficus bengalensis var - krishnae" "Araucaria Cookie" "Bombax Ceiba") ["Amherstia Nobilis" "Adansonia digitata" "Ficus bengalensis var - krishnae" "Araucaria Cookie" "Bombax Ceiba"]   user=> ["Amherstia Nobilis" "Adansonia digitata" "Ficus bengalensis var - krishnae" "Araucaria Cookie" "Bombax Ceiba"] ["Amherstia Nobilis" "Adansonia digitata" "Ficus bengalensis var - krishnae" "Araucaria Cookie" "Bombax Ceiba"]view rawvector_defn.clj hosted with ❤ by GitHubRetrieving the nth element of the vector is optimised for a vector. user=> (def plants-in-lalbagh ["Amherstia Nobilis" "Adansonia digitata" "Ficus bengalensis var - krishnae" "Araucaria Cookie" "Bombax Ceiba"]) #'user/plants-in-lalbagh user=>   user=> (nth plants-in-lalbagh 2) "Ficus bengalensis var - krishnae" user=> (get plants-in-lalbagh 2) "Ficus bengalensis var - krishnae" user=>view rawvector_lookup.clj hosted with ❤ by GitHubConcatenation works the same way to lists but conjoining happens at the end of the vector. user=> (def plants-in-lalbagh ["Amherstia Nobilis" "Adansonia digitata" "Ficus bengalensis var - krishnae" "Araucaria Cookie" "Bombax Ceiba"]) #'user/plants-in-lalbagh user=>   user=> (nth plants-in-lalbagh 2) "Ficus bengalensis var - krishnae" user=> (get plants-in-lalbagh 2) "Ficus bengalensis var - krishnae" user=> (conj plants-in-lalbagh "eucalyptus") ["Amherstia Nobilis" "Adansonia digitata" "Ficus bengalensis var - krishnae" "Araucaria Cookie" "Bombax Ceiba" "eucalyptus"] user=> (def plants-in-lalbagh1 ["Amherstia Nobilis" "Adansonia digitata"]) #'user/plants-in-lalbagh1 user=> (def plants-in-lalbagh2 ["Araucaria Cookie" "Bombax Ceiba"]) #'user/plants-in-lalbagh2 user=> (concat plants-in-lalbagh) plants-in-lalbagh plants-in-lalbagh1 plants-in-lalbagh2 user=> (concat plants-in-lalbagh1 plants-in-lalbagh) plants-in-lalbagh plants-in-lalbagh1 plants-in-lalbagh2 user=> (concat plants-in-lalbagh1 plants-in-lalbagh2) ("Amherstia Nobilis" "Adansonia digitata" "Araucaria Cookie" "Bombax Ceiba") user=>view rawvector_characteristics.clj hosted with ❤ by GitHubC. Maps or HashMaps.Maps are equivalent to dictionaries in Python and HashMaps in Java. Maps associate keys to values and the keys must be comparable. There are numerous implementations of maps with various guarantees of ordering and different performance on the lookups.You can create a map using literals.user=> (def fav-filmmakers {:stanley-kubrik “26 July 1928”#_=> :satyajit-ray “2 May 1921”})#’user/fav-filmmakersuser=>Note that keywords are used as keys here. This is more of a convention and not really required. To access the value corresponding to a particular key, you can use the get function or use the key specified as a function. user=> ;; keys as a function.   user=> (:satyajit-ray fav-filmmakers) "2 May 1921" user=> ;; using the get function.   user=> (get fav-filmmakers :satyajit-ray) "2 May 1921"view rawmap_lookup.clj hosted with ❤ by GitHubD. SetsSets are similar to mathematical sets and are a collection of unique values. They are hashed and unordered.To create a set you can use a literal or the set function.user=> #{1 2 3 4 5}#{1 4 3 2 5}user=> (set [1 2 3 4 5])#{1 4 3 2 5}Sets are collections so you can count, conjugate them with other elements or break them or see if they contain some element. user=> (def banned-books #{"the hindus", "The God of Small Things"}) #'user/banned-books user=> (count banned-books) 2 user=> (seq banned-books) ("The God of Small Things" "the hindus") user=> (conj banned-books "Five Past Midnight in Bhopal") #{"The God of Small Things" "Five Past Midnight in Bhopal" "the hindus"} user=> (disj banned-books "The God of Small Things") #{"the hindus"} user=> (contains? banned-books "the hindus") trueview rawset_as_collections.clj hosted with ❤ by GitHubAs you can expect from the name, sets support set-operations like union, difference, intersection, etc. Set operations require the set namespace to be loaded. user=> (require '[clojure.set :as set]) niL user=> ; set operations user=> (set/union #{1 2} #{2 3} #{1 3 2} user=> (set/intersection #{1 2} #{2 3}) #{2} user=> (set/difference #{1 2} #{2 3}) #{1}view rawset_operations.clj hosted with ❤ by GitHubNote:DataStructures are functions:Maps are functions of their keysKeywords are functions of mapsSets are functions of their elementsVectors are functions of their indicesHope you liked this!To get regular updates on more content, learning material and general rants you can follow me on Twitter.

Learn More >

Abstract Base Classes in Python, Redefine Future of Banking

Python is a dynamic and interpreted language. The result is that we do not know the errors in the code unless we run them. This is not the case with languages which are more static and compiled.Say, you are writing a new banking API. The business case requirement is quite mission critical and so, the library needs to be future-proof. You want the future developers, who are going to use the API, to be implementing the functionality in a specific way.Let’s say, you have your Banking class which will implement some methods. While creating a class, you are kind of making a promise that this class will implement some methods. To keep things simple, we will only take the case of one method execute_payment.class Banking(object):""" Payment processor that does nothing, just logs """def __init__(self):self._logger = logging.getLogger(__name__)self._logger.setLevel(logging.INFO)def process_payment(self, destination_address, amount):""" Execute a payment to one receiving single addressreturn the transaction id or None """print("process payment for bank.")Similarly, you have started writing implementations for various banks.class BankingBank1(object): ...class BankingBank2(object): ...Now, you see a pattern in them. Being a firm believer in the DRY (Don’t Repeat Yourself) pattern, you will probably move the similar methods to a base class and the child class will implement specific functionality.But, the subclasses may not be implementing some necessary functionality. You put that on the TODO list and go on a vacation for some days. Once back, you see code like this.class Banking(object):""" Payment processor that does nothing, just logs """def __init__(self):self._logger = logging.getLogger(__name__)self._logger.setLevel(logging.INFO)def process_payment(self, destination_address, amount):""" Execute a payment to one receiving single addressreturn the transaction id or None """raise NotImplementedError("Not implemented yet")class BankingBank1(Banking):"""docstring for BankingBank1"""def __init__(self, arg):super(BankingBank1, self).__init__()self.arg = argdef process_payment(self, destination_address, amount):""" Execute a payment to one receiving single addressreturn the transaction id or None """print('process payment for bank 1')class BankingBank2(Banking):"""docstring for BankingBank1"""def __init__(self, arg):super(BankingBank2, self).__init__()self.arg = argdef user_action(bb_obj):if isinstance(bb_obj, BankingBank1):bb_obj.process_payment('destination_address', 100)elif isinstance(bb_obj, BankingBank2):print('Cannot process the payment.')else:print('implement this.')bb2 = BankingBank2(__name__)user_action(bb2)You are appalled. There are multiple if statements checking the type of the object and then deciding the course of action based on that. Classic LBYL. You make a mental note about educating everyone in the team why LBYL is bad. A champ of the EAFP principle, you refactor the code.class Banking(object): ...class BankingBank1(Banking): ...class BankingBank2(Banking): ...def user_action(bb_obj):try:bb_obj.process_payment('destination_address', 100)except NotImplementedError as e:print('Cannot process the payment.')bb2 = BankingBank2(__name__)user_action(bb2)But you are not happy. The issue with this kind of implementation is that the object gets created and error is not raised till the method is called. This might be a problem if some other methods are getting executed before this method is called. This is a problem in interpreted languages like Python. In compiled languages, this kind of error would get raised at the compilation phase itself.Things should fail fast and if possible at the instantiation level itself, isn’t it? There goes your mentor, the programming friend who always helps out and lo, he tells you that it can be achieved using the abstract base class and the abstract method decorator. He sketches out the rough basics and you start writing the code.""" Contains payment processors for executing payments """from abc import ABCMeta, abstractmethodclass Banking(metaclass=ABCMeta):""" Payment processor that does nothing, just logs """def __init__(self):self._logger = logging.getLogger(__name__)self._logger.setLevel(logging.INFO)@abstractmethoddef process_payment(self, destination_address, amount):""" Execute a payment to one receiving single addressreturn the transaction id or None """passclass BankingBank1(Banking):"""docstring for BankingBank1"""def __init__(self, arg):super(BankingBank1, self).__init__()self.arg = argdef process_payment(self, destination_address, amount):""" Execute a payment to one receiving single addressreturn the transaction id or None """print('process payment for bank 1')class BankingBank2(Banking):"""docstring for BankingBank1"""def __init__(self, arg):super(BankingBank2, self).__init__()self.arg = argtry:bb2 = BankingBank2(__name__)bb2.process_payment('destination_address', 100)except TypeError as e:print('Cannot process the payment.')You test if it works or not. Sure enough, TypeError gets raised while trying to create the object itself.You can now rest assure that no functionality will get implemented and the object will fail in the instantiation phase only.You know a new pattern now and want to know some more before calling it a day. You go through various resources:Collections.abc has some common patterns that can be implemented. You can create custom types using these patterns.Some good design patterns are implemented using abstract method.References:Leonardo Giordani. 2016. “Abstract Base Classes in Python”.Stackoverflow. “Why use abstract base classes in Python”Guido van Rossum. 2007. PEP 3119.Dan Bader. Abstract Base Classes in PythonHope you liked this!To get regular updates on more content, learning material and general rants you can follow me on Twitter.

Learn More >

Understanding Method Definition in Python

The idea of method or function definition is probably language and domain-agnostic, i.e., more of a rational way of writing reusable codes and a clever way of implementing DRY (Don’t Repeat Yourself)A method definition in Python is as easy as in any other language. The functional block starts with a def keyword, followed by a function name and parentheses.def hello_world():"""Print Hello World"""print "Hello World">>>hello_world()"Hello World"So, how does a method work in Python? We will first look into the instance methods. Let’s start with a familiar example.class Animal:"""Animal class"""def __init__(self, name):"""Initialize animal"""self.name = namedef animal_name(self):"""Return animal name"""return "This is a {}".format(self.name)Here, we have a class called Animal with two methods __init__ and animal_name, where former is a special method and later is a plain pythonic instance method declaration. It can be seen that we have passed ‘self’ keyword in both these methods. What does this mean? Let us try to understand it with an example.>>> Animal.animal_name<unbound method Animal.animal_name>>>>Animal.animal_name()Traceback (most recent call last): File "<stdin>", line 1, in <module> Animal.animal_name()TypeError: unbound method animal_name() must be called with Animal instance as first argument (got nothing instead)The keyword ‘self’ is just a naming convention (other words can also be used). By providing ‘self’ as the first argument we are specifying that the methods within the class Animal are bound to the class instance, i.e, the first argument passed is itself the instance of a class. The below example will make it clearer.>>>animal = Animal('dog')>>>animal.animal_name<bound method Animal.animal_name of <__main__.Animal instance at 0x0216F9E0>>>>>animal.animal_name()'This is a dog'In this case, we create an instance of class Animal which is passed as the first argument to the method ‘animal_name’ or more precisely speaking Bound method which is same as doing:>>>Animal.animal_name(Animal('cat'))'This is a cat'Bound and Unbound methodsIn simple terms, a bound method is the one which is dependent on the instance of the class as the first argument. That’s why they are called instance methods and in our example, animal_name is an instance method. Staticmethods are examples of unbound methods. Speaking in terms of our previous example we can find an analogy, any animal_name is first bound to Animal first and then supposed to display its characteristics.StaticmethodIn simpler terms, staticmethods are those methods which are not bound to the class instance and deal only with the arguments it takes. To declare a staticmethod we just have to add ‘@staticmethod’ decorator over the method definition as shown in the next example.class Animal:"""Animal class"""def __init__(self, name):"""Initialize Animal"""self.name = namedef animal_name(self):"""Return animal name"""return "This is a {}".format(self.name)@staticmethoddef message():"""Return message"""return "Love animals!"We can see in the example that we have not added ‘self’ keyword in the message which suggests that it is not related to the class or its instance. Since it is not related to class anyway, it is not mandatory to call staticmethods with instance of class. Following example will give a clear idea.>>>Animal.message()"Love animals!">>>a = Animal('dog')"Love animals!">>>Animal.message <function __main__.message>Hence, we see that staticmethods are not bound to the class. Also, the analogy with the example is that the message (“Love animals”) is more of a generic type and not related to the class, so staticmethods are the ones which do not relate to class anyway. By using staticmethods, we are essentially trying to demonstrate that the method depends on arguments and does not touch the instance (i.e. ‘self’). We can see the implementation in datetime module of CPython repository.ClassmethodMuch like staticmethods, classmethods also are independent of the instance of class but are dependent on the class itself, i.e., bound to the class. Thus, quite obviously it takes class as the first parameter. Classmethods can be created with ‘@classmethod’ decorator as described in the next example.class Animal:"""Animal class"""animal = 'dog'@classmethoddef change_animal(cls, new_animal):"""Change animal for class"""return cls.animal = new_animaldef print_animal(self):"""Return animal"""return self.animalIn the above example, we are passing ‘cls’ as the class reference. Let’s see how this works.>>>Animal.animal'dog'>>>Animal.change_animal<bound method classobj.change_nimal of <class __main__.Animal at 0x7f63aefca9a8>>>>>Animal.change_animal('bird')>>>Animal.animal'bird'The example makes it clear that classmethods are bound to the class itself and thus, is powerful enough to change the class properties; in this case, the class is variable. Also, it is notable that much like staticmethods, these do not need to be called on class objects. Hence, classmethods are also called factory methods or alternate constructors. An example of alternate constructor is shown below:class Animal:"""Animal class"""def __init__(self, legs):"""Initialize animal"""self.legs = legs@classmethoddef change_legs(cls):"""Change legs for animal."""return cls(4)>>>animal = Animal(2)>>>animal.legs2>>>animal = Animal.change_legs()>>>animal.legs4An implemented example of classmethod can be seen here. Apart from classmethods and staticmethods, abstract methods are of great usage. You can read about abstractmethods and abstract base classes here. Happy Python!

Learn More >

Demystifying Basic Statistics in Machine Learning

These days there is a Cambrian explosion of various data science and machine learning tools that make it very easy to start in machine learning. Probably, you are someone who has heard about the buzzword and wanted to try it out yourself. Maybe you have gone through tutorials on one of the hot and trending machine learning libraries such as scikit-learn and want to have an idea on how to implement machine learning. You recognize that you have all the prerequisites of a problem that make it suitable for machine learning. You have the data set and also a problem that seems to have a pattern to it, but you cannot pin it down using an algorithm. You threw the data set to the machine learning library and got something as an outcome.Now, starts the problem. You don’t understand what this ‘something’ is. More often than not this ‘something’ is a collection of numbers in a weird format. Apart from better interpretability, the business for which your product or insights are built, wants to know if this is the best that can be done. As you are the one who implemented this, they want you to prove that this is the state of the art results and if not, it’s you who needs to improve the results. And secretly you are starting to realize that you have no clue what is happening around you.Machine Learning is a buzzword right now and there are many who would be trying their hands at it. Therefore, it would help you a great deal if you are able to understand a little bit about the intricacies that make machine learning tools and libraries so powerful. In this post, we will look at the basic statistical concepts that make up the bedrock of analysis in machine learning.Types of DataUnderstanding different types of data will help you in choosing the different types of techniques that you may use to get insights into the data.The major types of Data are:Numerical:This represents some quantifiable thing that you can measure. This can be discrete, which is normally the counts of some event. As for example, how many times a person visits the same doctor when he is sick, or the number of clothes bought by a customer on an eCommerce platform. Another flavor is that the numerical data can be continuous as well. A characteristic of continuous data is that the range can be infinite. An example maybe, how much rainfall happened in a given year and so on.Categorical:These are data that has no inherent numerical meaning, such as man, woman. Or the state of birth of a group of people and so on. Generally, numbers are assigned to these values but the numbers themselves don’t mean anything. One of the good ways to denote categorical values is through graphs. So, in the figure, the scientists ‘Chandrasekhara Venkata Raman’ and ‘Jagadish Chandra Bose’ both have the birthplace as India which is shown by the edges.Ordinal: This is the mixture of numerical and categorical data. An example is the ratings given by a customer as in 5 stars is better than 1 star.Mean Median ModeThese are the measures of central tendency of a data set.sourceMean is given by the total of the values of the samples divided by the number of samples. In mathematical terms this would mean:where x̄ is the mean and there are n values of x. Σ means that you need to take the sum of all the data points.Below is an example where the mean of a set of data points is calculated.This is a simple example taken to explain discrete data. The mean can be calculated in case of continuous data as well.To calculate the median, sort the values and take the middle value. To illustrate let x denote the following numbers:x = [23, 40, 6, 74, 38, 1, 70]In this case, you would need to sort the list first and then take the midpoint to find the median.sorted_x = [1, 6, 23, 38, 40, 70, 74]In this case, the midpoint is 38 since the value 38 divides sorted x into 3 values which are smaller than 38 and three values which are higher than 38. Now, in case there are even number of values then the average of the two middle values are taken as the median. The advantage of the median over the mean is that median is less susceptible to outliers. So, in situations where there is a high chance that there may be outliers present in the data set, it is wiser to take the median instead of the mean. For example, to understand what is the per capita income of a country the median is taken because the rich may be extremely rich which would skew the average and show a different picture than what the average people might experience. In that case, probably, taking the median would give a better insight.Mode represents the most common value in a data set. Mode is most useful when you need to understand clustering or number of ‘hits’. For example, a retailer may want to understand the mode of sizes purchased so that he can set stocking labels optimally. Say, store A has a mode of ‘small’ while store B has a mode of ‘XXL’.In this post, we looked at some of the basic statistics that you will encounter while looking at data for machine learning. There are some more concepts that you need to know and so head over to my next post.Also, if you are interested in talking more on this, just drop me a message @alt227Joydeep. I would be glad to discuss this further. You can also hit the like button and follow me here, on Medium.References:Quora. Is statistics a prerequisite for machine learning.Understanding different data types.HackerEarth. Descriptive Statistics.Draw a skewed normal dist.

Learn More >

How can Intelligent Chatbots Make the Brand User Friendly?

Chatbots and Virtual Assistants are slowly making their way towards becoming the modern consumer’s first touchpoint towards interaction with a brand. From providers of information to reliable automation that assists consumers with their queries to become the front face of the company, Chatbots have evolved significantly over the years, making strides towards intelligence and self-governance.In addition to being automation capable of solving problems on a real-time basis, chatbots have also become an incredible tool in engaging and keeping a consumer interested in the brand. Unique chatbot experiences are sought and cherished by its users.There are a number of strategies that can be employed to significantly increase chatbot engagement. Two of the most common ways for going about this is to either create a sense of curiosity in the consumer or create touchpoints that surprise and delight the end-user. Both these ways significantly increase chatbot usage and often serve as the foundation on which other strategies are integrated. Here are a few other ways you can make your chatbots more interesting.Build Chatbots that learnA chatbot that is personalized to suit the needs of the user it is talking to is always preferred to a chatbot that seems distant. The trick here is to surprise and delight the consumers by allowing the chatbot to learn from its conversations and adopt a pattern that is more likable to the user. One of the best strategies you can adopt is measuring the retention rate of the chatbot. Since the chatbot learns, adapts, and reacts in certain ways to certain target groups, it is possible to analyze these chatbots with higher retention rates and obtain the factors that would allow you to create a unified and delightful experience for all your consumers.Remove MonotonyThe most common complaint that chatbots receive is about their repetitive and stale experience every time a consumer gets a chance to interact with them. Chatbots are expected to learn and improve to offer a more refined yet fresh experience every time the user comes in touch with them. A chatbot that is centered around ease and comfort with a pinch of fresh engagement can go a long way in removing the monotony that users often find themselves complaining about. Intermixing various interactive options such as gifs, emoticons, banter, and quips in between wait times, suggestions, or predictions of what the user might also like are strategic ways that can guarantee positive engagement. A chatbot’s flow of conversation must always inject a feeling of curiosity into the user’s minds and make them wonder what else said bot can do.Gamification of FunctionsGamification of your chatbot can be a powerful incentive if used in the right way. It also relies heavily on the purpose that your bot has been created to achieve and provide entertainment or information that might be of use to the consumer. If the main purpose of your chatbot is to provide information or instill habitual patterns to its users, then gamification might be an added bonus to go about it.Gamification can be as complex as a quiz based on the previous information to reap rewards such as virtual medals, certificates, or even gift coupons. They can be as simple as giving the user a celebratory GIF. Another good strategy would be the ‘claiming’ of daily rewards or the announcement of special, limited bonuses only available during a very short period of time. Other forms of gamification could include the display of a competitive scoring of friends, point-based level up, and a reward system. The goal here is to ultimately make the consumer keep coming back for more.EncouragementAnother brilliant strategy to increase engagement is to allow the chatbot to encourage the user’s to spend more time with it. This can be achieved through prompts and questions on the topic of conversation that would naturally pique the user’s curiosity. Chatbots that deliver seasonal, or daily content can end with a tease or short glimpse of the next day’s content to influence the return of the user or consumer. Chatbots can also suggest similar topics of conversation to the user to continue to engage them.HumanizationHumanizing the chatbot is one of the best ways to instill a sense of familiarity and trust with the users. A study by eMarketer revealed that respondents always preferred to communicate with a real person in a choice between a chatbot and the latter. Building a chatbot interface with human-like characteristics: name, display image would make the consumers feel like they are talking to a real person and would overall increase user engagement.Grab the User’s Attention with Push NotificationsA chatbot or virtual assistant can be a game-changing asset in building a user-friendly brand. But it can only be helpful if the visiting users notice and use it.Push Notifications, whether on mobile or desktop browsers will help grab the attention of users almost immediately. A popup hook accompanied by a notification sound may help redirect the user’s attention towards the chatbot and therefore prompt them into using it to help assist them better.

Learn More >

Deep Dive into Jupyter Notebooks to Master ML Analysis

In case you stumbled on my blog series, here’s a brief about my blogging activities. I make deep dives on the important machine learning algorithms. After completing an overview of machine learning I had started with linear regression. Now before moving on, I want to take a kind of break from the flow and discuss the tooling that an everyday machine learning engineer uses. In fact, I would be specifically talking about the most important tool that he/she would need in their day to day work: Jupyter Notebook. If you have been interested in machine learning at least for some time then I am sure you must have heard of this tool and actively using it. Through this post, I am hoping that I will be able to shed light on some interesting ways in which you can use jupyter notebooks to take your analysis to the next level.HistoryBut first, let’s look at the origin of this awesome software. In the data-science field as well as the broader scientific community, the kind of analysis that has been resorted to has been largely exploratory. Not only is the output important but the methodology and the way the output is generated is also important. Previously it was difficult to share the line of thinking, hence, there was always a need for such a tool which facilitates it. Therefore, the IPython Notebook was developed in 2011 by a team of researchers under Fernando Pérez and Brian Granger. It was then called IPython because the kernel could only execute Python code. In 2014 the Project Jupyter started as a spin-off project from the IPython project. In the new notebooks, the kernels were separate from the frontend and support for other languages apart from Python was brought in.Just as a side note, Jupyter is an acronym for Julia, Python and R, signifying that the project creators want to encompass the whole sky of machine learning under its umbrella and provide the power of lightning and flash to all its users.:)Version ControllingOnce you start maintaining jupyter notebooks for reproducible results, it is important to have an effective storing mechanism. Github is an excellent platform to store and view the notebooks as it renders the notebook when you open the respective file link on any web browser. There are two caveats though. First, is if you have a lot of JavaScript in your notebook, then they are not rendered in Github. You will then need to see them on your local machine. Other than that sometimes differences in code are tough to see and hence, the changes with the previous code are pretty hard to see due to the file structure of the ipynb files.The solution is to save python files with the jupyter notebook so that the differences in code can be tracked easily as well. This can be done by editing the config file:~/.ipython/profile_nbserver/ipython_notebook_config.py ### If you want to auto-save .html and .py versions of your notebook: # modified from: https://github.com/ipython/ipython/issues/8009 import os from subprocess import check_call   def post_save(model, os_path, contents_manager): """post-save hook for converting notebooks to .py scripts""" if model['type'] != 'notebook': return # only do this for notebooks d, fname = os.path.split(os_path) check_call(['ipython', 'nbconvert', '--to', 'script', fname], cwd=d) check_call(['ipython', 'nbconvert', '--to', 'html', fname], cwd=d)   c.FileContentsManager.post_save_hook = post_saveview rawipython_notebook_config.py hosted with ❤ by GitHubThis would save a .py and .html file with each notebook saved. Then the differences in the code can be easily shown in the console.Saving images as a separate file also has a lot of benefits as you can compare the differences between the two. Click on this link and see how the differences in the images can be easily shown.Two kinds of notebooksSince jupyter notebooks can be used as an exploratory tool as well as a final document that can be delivered and shown to the customer, it is highly advisable for analytics teams to have two kinds of folders for a specific use case. One can be named develop where exploratory analysis can be kept while the other can be named deliver to house the final notebooks. So you will have the following folder structure.– develop– ––+ analysis1.ipynb+ analysis1.py+ somefig.png– deliver–+ final_analysis.ipynb+ final_analysis.py+ fig1.pngMagicsThere are a lot of magics that can be used in jupyter notebooks. A listing of all the magics will make this post too long, hence, will only focus on the ones that I find most useful in my day to day work.The first one is using the magic bash. Bash commands make it easy to see the files in the current directory and also download files and other simple bash commands without leaving the comfort of the browser. You can also show the state of the directory where you are probably downloading some file and would like to show that the file is present in some remote directory like s3 and you will need to download the file.view rawbash_magics.ipynb hosted with ❤ by GitHubAnother one I like is showing math equations. Since we as data scientists work with a lot of algorithms, we need to articulate the algorithms that we use for other engineers who we are working with. Hence %%latex magic really helps us in that manner.view rawlatex.ipynb hosted with ❤ by GitHubStructuring a Jupyter NotebookIn the beginning, there should be the imports and the dependencies. For Python, imports should be ordered in the normal PEP8 way. i.e.,1.standard libraries2.related third-party imports3.local application/library/module importsNext, the version information should be given on at least the main dependencies like numpy, scipy, pandas etc.view rawstarting_notebook.ipynb hosted with ❤ by GitHubOnce you have reached near the end of your notebook you should consider two things.Is my notebook too big? In case its too big, then probably it will not be clear to the reader what the big picture is. Hence, probably you should consider breaking the notebook into separate notebooks. To do that you can take help of the FileLinks function.view rawlinking.ipynb hosted with ❤ by GitHubJupyter Notebooks are meant to be stories about data and hence should have a conclusion summarizing the main points and what has been discussed.DebuggingIf the code that you are trying to debug lands in the realm of pure python, then you can employ the age-old python debugging tools of pdb. The pdb commands and methods work in the jupyter notebook as well.view rawdebugging_using_python.ipynb hosted with ❤ by GitHubIn my experience pdb should help you a great deal while debugging your models. This is based on the assumptions that you create your models in python. In case you are creating your models in some other language then you will need to use the debugging tools for the respective language, but I am going to assume that jupyter would work the same way in case the language is an interpreted one like R, Julia, etc. In case you are finding some other kind of errors that look more like weird matrix issues, then probably the shape of the data that you are passing is wrong and you need to look closer at how you are building your matrices.ProfilingSince you will be dealing with huge amounts of data, it is a good thing to know such things as memory consumption, time spent in executing the cells, especially the cells that are doing most of the heavy lifting, and so on.%%time will tell you the amount of time needed to run the cell.view rawprofiling_time.ipynb hosted with ❤ by GitHubYou can also use line profiler or see how much memory the cells are taking. For that, you will need to install line_profiler and memory_profiler depending on your requirement. Then you can load the extensions.view rawprofiling_line_memory.ipynb hosted with ❤ by GitHub

Learn More >

Prolog Logic Programming: A Detailed Explanation

Prolog logic programming language is a general-purpose programming language. If you have never heard of this language and are a SQL aficionado, you will recognize the way of writing code in this language, which feels quite like a query. This is a declarative language or a rule-based language like SQL. Note that in “normal” languages that you encounter, like C, Java, and Python, you generally write out the ways in which you retrieve the data or do the computation. Here, you will just be defining what you want instead of writing out explicitly how to get it. Feels wonderful, isn’t it? Now, since this is a vast language by itself, only a minuscule part will be looked at.1.install swipl2.take a look at some basic examples to have some idea on how prolog code looks like3.ruminate on some basic ways in which we can reason about codeSWIPLSWI-Prolog is one of the most comprehensive implementation for prolog. It is also very widely used. You can use the SWI-Prolog interpreter to run the programs. To install the interpreter in ubuntu you can run the following commands.sudo add-apt-repository ppa:swi-prolog/stablesudo apt-get updatesudo apt-get install swi-prologNow, create a new hello.pl and type the below lines: /* * usage: swipl -q -f prolog_hello.pl -t main */ main :- write("Hello world!"), nl, fail.view rawprolog_hello.pl hosted with ❤ by GitHubYou have created a hello world program for prolog. In the terminal, run the following command to run this file and you should get the output in the terminal.➜ swipl -q -f hello.pl -t mainHello WorldSome code examplesTake a look at some code samples to understand the flow of logic in a prolog code. To show relationships between data, write in the following format.verb(noun1, noun2)which means that there is a relationship from noun1 to noun2 via relationship verb . Using this define the following relationships. has(jack,apples). % jack has apples. has(jack,plums). % jack has plums. has(ann,plums). % ann has plums. has(dan,money). % dan has money. fruit(apples). % apple is fruit. fruit(plums). % plums is fruit.view rawprolog1.pl hosted with ❤ by GitHubThis is logic programming where the facts and rules of the problem domain is defined. The following code will list the members of fruit class.https://gist.github.com/infinite-Joy/infinite-Joy/0cb2efac38507593cd0380e6d504e2aeThis is called a query. Save this to a listing.pl file and then execute the query. The command is written as:swipl -q -f -t➜ swipl -q -f listing.pl -t mainfruit(apples).fruit(plums).This shows a listing of all the relationships in the data.Similarly, based on the existing relationships, write the names of the people who have apples or plums. main :- has(X,apples), % who has apples has(Y,plums), % who has plums write(X), % write the apple lovers put(10), % new line write(Y), % write the plum lovers nl, halt. % new line and haltview rawprolog3.pl hosted with ❤ by GitHubRun the file in the terminal using the interpreter➜ prolog swipl -q -f listing1.pl -t mainjackannYou can find the whole file in this github gist.Now as an exercise can you find the people who have both apples and plums. Share the gist in the comments below.In case you found this interesting and would like to talk more on this, just drop me a message @alt227Joydeep. I would be glad to discuss this further. You can also hit the like button and follow me here, on Medium.

Learn More >