Menu

Blog

Explore our insightful blog posts that delve deeper into the latest industry trends, best practices, and success stories. Gain valuable knowledge and stay informed about the ever-evolving landscape of digital transformation.

Scala XML Data Processing: Efficient Spark Pipelines

As data scientists and machine learning engineers, we don’t always appreciate that most of the data we get are usually in CSV or at times in JSON file format. In reality, this is great, as we need to deal with large volumes of data and any format that makes it easy to read and understand data should be highly appreciated. And people who work with CSV know how great this is as a data format. In this blog we will be focusing on Scala XML data processingHaving said that, it might not always be the case. If you are a Scala developer (a JVM language), you are likely to work in a Java environment. And since XML has been the preferred format for data interchange, you are most likely to receive data in an XML format. Which means, you will need to parse the data from XML files and build data pipelines out of it.XML, which stands for Extensible Markup Language, was thought of as a way in which both computers and humans should be able to understand the text. Of course, the designers got their inspiration from the hugely successful HTML. You might argue that no one actually reads HTML and we only see the final output thrown by the browsers. Well, may be, it was assumed that XML would be read only by developers and hence it should work. But then we moved to Service Oriented Architecture (SOA) where XML has become the standard data-format for communication between services. In this post we will see how we can parse XML in spark-Scala.<?xml version="1.0"?><!DOCTYPE PARTS SYSTEM "parts.dtd"><?xml-stylesheet type="text/css" href="xmlpartsstyle.css"?><PARTS> <TITLE>Computer Parts</TITLE> <PART> <ITEM>Motherboard</ITEM> <MANUFACTURER>ASUS</MANUFACTURER> <MODEL>P3B-F</MODEL> <COST> 123.00</COST> </PART> <PART> <ITEM>Video Card</ITEM> <MANUFACTURER>ATI</MANUFACTURER> <MODEL>All-in-Wonder Pro</MODEL> <COST> 160.00</COST> </PART> <PART> <ITEM>Sound Card</ITEM> <MANUFACTURER>Creative Labs</MANUFACTURER> <MODEL>Sound Blaster Live</MODEL> <COST> 80.00</COST> </PART> <PART> <ITEMᡋ inch Monitor</ITEM> <MANUFACTURER>LG Electronics</MANUFACTURER> <MODEL> 995E</MODEL> <COST> 290.00</COST> </PART></PARTS>Table 1 A simple XML file (1)Interestingly it’s quite easy to parse and create XML pipelines in Scala. To load an XML file, you will need to pass the filename to the loadFile utility from XML. Please note that parsing the whole file requires a lot of processing power and therefore, chances are that you may run into ‘OutOfMemoryError’ as shown in table 2.scala> import scala.xml.XMLimport scala.xml.XMLscala> val xml = XML.loadFile("data/Posts.xml")java.lang.OutOfMemoryError: Java heap space at java.util.Arrays.copyOf(Arrays.java:3332)...scala.runtime.ScalaRunTime$.replStringOf(ScalaRunTime.scala:345) at .$print$lzycompute(<console>:10) at .$print(<console>:6) at $print(<console>) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)Table 2 Scala code first runIn case that happens, you will need to boost the memory for the spark driver. I am allocating a random high number here (as shown in table 3).spark-shell --driver-memory 6GTable 3 Increased driver memoryNow I can parse the XML with ease (refer to table 4).scala> import scala.xml.XMLimport scala.xml.XMLscala> val xml = XML.loadFile("data/Posts.xml")xml: scala.xml.Elem =<posts> <row FavoriteCount="1" CommentCount="1" AnswerCount="1" Tags="&lt;job-search&gt;&lt;visa&gt;&lt;japan&gt;" Title="What kind of Visa is required to work in Academia in Japan?" LastActivityDate="2013-10-30T09:14:11.633" LastEditDate="2013-10-30T09:14:11.633" LastEditorUserId="2700" OwnerUserId="5" Body="&lt;p&gt;As from title. What kind of visa class do I have to apply for, in order to work as an academic in Japan ? &lt;/p&gt;" ViewCount="415" Score="16" CreationDate="2012-02-14T20:23:40.127" AcceptedAnswerId="180" PostTypeId="1" Id="1"/> <row ClosedDate="2015-03-29T20:06:49.947" CommentCount="2" AnswerCount="2" Tags="&lt;phd&gt;&lt;job-search&gt;&lt;online-resource&gt;&lt;chemistry&gt;" Title="As a computational chemist, which online resources are avail...Table 4 Read XMLI have not talked about the file data that I am using till now. The xml dataset is a stackoverflow dataset downloaded from archive.org. It contains data in this format.<?xml version="1.0" encoding="utf-8"?><posts> <row Id="1" PostTypeId="1" AcceptedAnswerId="180" CreationDate="2012-02-14T20:23:40.127" Score="16" ViewCount="415" Body="&lt;p&gt;As from title. What kind of visa class do I have to apply for, in order to work as an academic in Japan ? &lt;/p&gt;&#xA;" OwnerUserId="5" LastEditorUserId="2700" LastEditDate="2013-10-30T09:14:11.633" LastActivityDate="2013-10-30T09:14:11.633" Title="What kind of Visa is required to work in Academia in Japan?" Tags="&lt;job-search&gt;&lt;visa&gt;&lt;japan&gt;" AnswerCount="1" CommentCount="1" FavoriteCount="1" /> <row Id="2" PostTypeId="1" AcceptedAnswerId="246" CreationDate="2012-02-14T20:26:22.683" Score="11" ViewCount="725" Body="&lt;p&gt;Which online resources are available for job search at the Ph.D. level in the computational chemistry field?&lt;/p&gt;&#xA;" OwnerUserId="5" LastEditorUserId="15723" LastEditDate="2014-09-18T13:02:01.180" LastActivityDate="2014-09-18T13:02:01.180" Title="As a computational chemist, which online resources are available for Ph.D. level jobs?" Tags="&lt;phd&gt;&lt;job-search&gt;&lt;online-resource&gt;&lt;chemistry&gt;" AnswerCount="2" CommentCount="2" ClosedDate="2015-03-29T20:06:49.947" />Table 5 Our target datasetEach record is allocated a row tag, that combines multiple attributes. We can now parse these individual tags and get the value of the attributes. To parse the records, you will need to search an XML tree for the required data, using XPath expressions. The way it works is that you need to pass \ and \\ methods for the equivalent XPath / and // expressions.For example, you can get the ‘row’ tags and then on each record get the ‘body’ attribute. This gives us a sequence of scala.Option.scala> val texts = (xml \ "row").map{_.attribute("Body")}texts: scala.collection.immutable.Seq[Option[Seq[scala.xml.Node]]] =List(Some(&lt;p&gt;As from title. What kind of visa class do I have to apply for, in order to work as an academic in Japan ? &lt;/p&gt;), Some(&lt;p&gt;Which online resources are available for job search at the Ph.D. level in the computational chemistry field?&lt;/p&gt;), Some(&lt;p&gt;As from title. Not all journals provide the impact factor on their homepage. For those who don't where can I find their impact factor ?&lt;/p&gt;), Some(&lt;p&gt;I have seen many engineering departments want professional engineer registration. Why do they care? &lt;/p&gt;), Some(&lt;p&gt;What is the h-index, and how does it work ?&lt;/p&gt;), Some(&lt;p&gt;If your institution has a subscription to Journal Citation Reports (JCR), you...Table 6 Getting the appropriate textNow that you have an iterator you can run complex transformations on top of it.Below (in table 7) we are converting texts to string, trimming them for extra whitespace, then filtering out the text with some string, and converting them to lowercase.scala> val lower_texts = texts map {_.toString} map { _.trim } filter { _.length != 0 } map { _.toLowerCase }lower_texts: scala.collection.immutable.Seq[String] =List(some(&lt;p&gt;as from title. what kind of visa class do i have to apply for, in order to work as an academic in japan ? &lt;/p&gt;), some(&lt;p&gt;which online resources are available for job search at the ph.d. level in the computational chemistry field?&lt;/p&gt;), some(&lt;p&gt;as from title. not all journals provide the impact factor on their homepage. for those who don't where can i find their impact factor ?&lt;/p&gt;), some(&lt;p&gt;i have seen many engineering departments want professional engineer registration. why do they care? &lt;/p&gt;), some(&lt;p&gt;what is the h-index, and how does it work ?&lt;/p&gt;), some(&lt;p&gt;if your institution has a subscription to journal citation reports (jcr), you can check it t...Table 7 Spark transformationScala offers a convenient and easy way for basic XML processing. This post is aimed at helping beginners use XML and Scala with ease. If you found this useful, do leave a comment, we would love to hear from you and share the post with your friends and colleagues.

Learn More >

Optimizing Lean Agile Process: A Retrospective Approach

Lean agile process is a very effective way to handle any project. The methodology helps eliminate unnecessary processes, reducing ambiguity and allowing the team to utilize their time optimally in developing an effective software with better functionality. With lean agile process, team is better equipped to focus on objectives, improve productivity and deliver superior value to the customers.Few of the building blocks for implementing lean agile principles are grooming the user stories and the functionalities or epics by understanding acceptance criteria, prioritizing, and conducting retrospectives. While grooming is an important part of the process, retrospective is equally imperative that enables the team to understand what went well and what did not. The key objective of retrospective meeting is to understand the drawbacks of the process and make necessary changes. However, there’s no right or wrong way for retrospection.RetrospectionIn this post I am sharing our approach to retrospective meeting that helped my team during the development process. My team is a standard scrum team comprising software developers, test engineers, data engineers and analysts.The initiationWe followed a process where the first two sprints were development sprints and we started testing from the third sprint. While we conducted retrospective meeting after the end of each sprint, most of it looked good when the code was not tested.Retrospective meeting led to greater discussions from the third sprint onwards. We also had to dig deeper into the first sprint as the test engineers had more insights about their findings from the sprint-1. This resulted in code re-work in order to fix the bugs altering the velocity of the following sprints.Initial process included:Sprint-1: DevelopSprint-2: DevelopSprint-3: Develop + Test + Bug fixesSprint-4: Develop + Test + Bug fixesDepending on the severity of the bugs, the velocity varied from sprint to sprint because the priority of the client differed. In some sprints, we were expected to fix key functionality bugs first whereas in certain sprints we had to complete coding on priority.The issue with this process was that every time the client engineers produced better results than we did with the same set of data. While customer expected better results in each demo, test engineers were out of pace and didn’t have the clarity on what test cases to write.To address this challenge we decided to make some changes to the process in order to produce better outcome. The changes made to the process include:As soon as we receive the requirements, developers would make assumptionsThese assumptions need to be discussed in sprint meeting at the beginning of the sprints with the team and key stakeholders. This helps clarifying as many doubts as stakeholders may have right at the start of the sprint.Project managers to do risk analysis based on these assumptionsTest engineers to get involved right from the Sprint-1. Developers should provide the expected data output according to their assumptions to test engineers for them to write relevant test cases.Not to conduct demo of our work on fixed days but showcase when we develop or reach a significant stage in our model/approach.During the sprint, whenever we have any doubt in our assumption, discuss it with the related stakeholders because as we know, no result is wrong with respect to data or data science.Developers should perform unit testing followed by moving the code into test environment and during the testing process, developers should optimize the code (which is an important step added to the process).End of each sprint, justify the assumptions made, to the stakeholders and demonstrate the outcome.Here is an example of the scenario where we implemented the preceding changes. The code that we were writing for one of the sprints had to perform the following:Fetch 100 million records by reading line by line and channelize the data into a platform — Each record took 50 bytes (5 kb in total) and it took 2.3 minutes to fetch all the recordsOn the platform we cleanse the data and perform some mathematical operations through an API in order to produce the predictive algorithmsWe were using certain streams in order to work with the data that included:InputStreamReader: It fetches data from the coding platform, converts byte into character and passes onto BufferedReaderBufferedReader: It reads the character streams from InputStreamReader and deploy the data sequence into the platformNew process implementationThe change in the process helped us in several ways. As we made assumptions we also provided test cases to the test engineers right at the beginning. This made it easy for them to understand the expected output and write better test cases while clarifying doubts as and when required helped optimize time and effort. We conducted the demo every time we reached a significant stage improving our client’s experience. On the other hand, as we kept our key stakeholders informed throughout the process making it easy to justify our assumptions in retrospective meetings.Apart from this, we accelerated the process of fetching data by optimizing code. The changes made were as follows:1. We improvised the way we read data from files using specific lightweight streams, which consumes less memory (such as FileInputStreams instead of InputStreamReaders)2. Made single statement query and set the connection autocommit to false3. Unit test results showed us that they were unnecessary code which increased the timelapse and we could avoid the boundary conditions which were not neededWe also deployed coding into the test environment. While the test engineers tested the code, developers started to optimize the code further. We re-designed the architecture — fetched first 1 million records using optimized SQL and then fetched next chunk in parallel with different set of code. We also used Byte channels to read the data and during the process we continually cleared the buffer in the code which helped boost the speed.Lean agile methodology for continuous improvementWe incorporated these changes at the optimization stage while testing was done parallelly that enabled us to save time and improve efficiency. It helped reduce coding time by 70% (it took total 69 seconds to fetch the entire datasets) and man hours by 35%. In addition to this, we built a cross functional team that allowed us to think from various perspectives, facilitating knowledge sharing and enhancing productivity.Combining lean and agile principles helps take advantage of key elements of both the methodologies and customize them to meet project-specific requirements. It allows to integrate continuous improvement and follow best development practices enabling the team to optimize performance and deliver value.Let us know how you liked the post and share your experience of using lean agile methodologies, we love to hear from our readers.

Learn More >

Exploring Key Machine Learning Algorithms

Machine learning algorithms serve as the bedrock upon which predictive models are built. They are akin to a toolkit comprising various tools—each designed for specific tasks and yielding distinct outcomes. Broadly categorized into three types: Supervised Learning, Unsupervised Learning, and Reinforcement Learning, these algorithms encapsulate diverse methodologies tailored to address distinct challenges.Supervised learning involves training a model with input features and corresponding output labels. Conversely, unsupervised learning delves into datasets lacking predefined labels, aiming to unveil concealed patterns or relationships. Reinforcement learning, on the other hand, revolves around optimizing parameters based on feedback loops, a process reminiscent of trial and error.Within this paradigm, several prominent algorithms stand out, each with its unique characteristics and applications. Let's delve into a brief overview of a few noteworthy ones:Ordinary Least Squares Linear Regression: Linear regression endeavors to establish a line that best fits the distribution of data points within a training set. This method, also known as Ordinary Least Squares, minimizes the sum of squared residuals. It extends seamlessly to multidimensional datasets, where the line transforms into a higher-dimensional plane.Logistic Regression: Despite its name, logistic regression operates as a classification technique. Unlike linear regression, it doesn't presume a linear relationship between variables. Instead, it assumes a linear dependence on the logit of independent variables, resulting in a linear decision surface.Support Vector Machines (SVM): SVM, a supervised learning algorithm, tackles both classification and regression tasks by delineating data points within an N-dimensional space. It identifies a hyperplane to segregate these points, making it ideal for high-dimensional datasets. However, its computational complexity escalates with dimensionality.K-means Clustering: K-means clustering endeavors to partition data into K clusters, each centered around a centroid. This algorithm, reminiscent of creating stereotypes among groups of individuals, iteratively refines centroids until convergence.Decision Trees: Decision trees epitomize a classifier structured in a tree format, traversed from the root to leaf nodes, ultimately revealing target values. With their innate ability to mimic human decision-making, decision trees offer intuitive insights. However, it's imperative to strike a balance, as overly complex trees may sacrifice accuracy.These algorithms constitute a foundational knowledge base for burgeoning machine learning enthusiasts. Much like tools in a craftsman's arsenal, they boast distinctive merits and demerits, rendering them applicable across diverse scenarios.

Learn More >

Mastering Machine Learning Basics

Model: A machine learning model can be a mathematical representation of a real-world process. To generate a machine learning model you will need to provide training data to a machine learning algorithm to learn from.Algorithm: Machine Learning algorithm is the hypothesis set that is taken at the beginning before the training starts with real-world data. When we say Linear Regression algorithm, it means a set of functions that define similar characteristics as defined by Linear Regression and from those set of functions we will choose one function that fits the most by the training data.Training: While training for machine learning, you pass an algorithm with training data. The learning algorithm finds patterns in the training data such that the input parameters correspond to the target. The output of the training process is a machine learning model which you can then use to make predictions. This process is also called “learning”.Regression: Regression techniques are used when the output is real-valued based on continuous variables. For example, any time series data. This technique involves fitting a line.Classification: In classification, you will need to categorize data into predefined classes. For example, an email can either be ‘spam’ or ‘not spam’. Target: The target is whatever the output of the input variables. It could be the individual classes that the input variables maybe mapped to in case of a classification problem or the output value range in a regression problem. If the training set is considered then the target is the training output values that will be considered.Feature: Features are individual independent variables that act as the input in your system. Prediction models use features to make predictions. New features can also be obtained from old features using a method known as ‘feature engineering’. More simply, you can consider one column of your data set to be one feature. Sometimes these are also called attributes. And the number of features are called dimensions.Label: Labels are the final output. You can also consider the output classes to be the labels. When data scientists speak of labeled data, they mean groups of samples that have been tagged to one or more labels.Overfitting: An important consideration in machine learning is how well the approximation of the target function that has been trained using training data, generalizes to new data. Generalization works best if the signal or the sample that is used as the training data has a high signal to noise ratio. If that is not the case, generalization would be poor and we will not get good predictions. A model is overfitting if it fits the training data too well and there is a poor generalization of new data.Regularization: Regularization is the method to estimate a preferred complexity of the machine learning model so that the model generalizes and the over-fit/under-fit problem is avoided. This is done by adding a penalty on the different parameters of the model thereby reducing the freedom of the model.Parameter and Hyper-Parameter: Parameters are configuration variables that can be thought to be internal to the model as they can be estimated from the training data. Algorithms have mechanisms to optimize parameters. On the other hand, hyperparameters cannot be estimated from the training data. Hyperparameters of a model are set and tuned depending on a combination of some heuristics and the experience and domain knowledge of the data scientist.These are some of the machine learning basics that I thought are important and should be looked into for someone who is a machine learning beginner. In case you have encountered some common terms which are not included here, do write about them in the comments below.Thanks for reading this. If you are interested in talking more on this, just drop me a message @alt227Joydeep. I would be glad to discuss this further. Also please hit the claps and help this article reach more audience.Reference:Difference between machine learning model and algorithm.Difference between labeled and unlabeled data.label and feature.what is a feature.overfitting and underfittingdimension, attributes and features.parameter and hyper parameter.stats.stackexchange. regularization.regularisation.explained

Learn More >

What Is Machine Learning?

Man has always been enamored and mystified by human intelligence. The ability to learn and understand patterns in seemingly unrelated things has contributed a great deal to our success here on this planet. Researchers have always been on the hunt for the holy grail, the ability to replicate this intelligence in a machine and, in the process, understand what is it that makes the human mind what it is. In a series of seminal papers Alan Turing, who is widely considered to be the father of theoretical computer science and artificial intelligence, gave the basics of the Turing machine, which is one of the first formal definitions of the general purpose computer. The logical conclusion after that was if the computer could think and this gave birth to the Turing Test. These thoughts on general computing have influenced most of the general purpose languages that we use today.When do we need machine learning?But, this has not been enough. Though general purpose computer languages are Turing Complete, this does not solve the whole problem of human intelligence as the computer can only solve those problems for which the algorithm is known. What if the algorithm is unknown? This is where machine learning algorithm comes in. Let’s take an example. Jack and Mary are a newlywed couple and they want to buy a place in the near suburbs where they want to raise a home. Investing in a home and buying a house is one of the most expensive and largest purchases of any person. So, Jack and Mary want an inexpensive way to monitor the price of the land that they are interested in. Now, the price of land depends on various factors like the area of the land and the locality it’s placed in, for which we need a real estate expert to evaluate this correctly. However, the expert may have biases when he is advising us. How can we know that we can trust the person? Only if we had an algorithm or a mathematical formula that could correctly answer, that would be perfect. This is a kind of problem that probably does not seem to have anything like that. So, in this case, we will take the help of machine learning.Please note that some may argue that the theory explained here falls under the subcategory of supervised learning under machine learning and does not show all the capabilities of machine learning. Having said that, the aim of this post is not to show all the facets of machine learning but for the reader to have some “idea” or “feel” of what machine learning is all about and how machine learning happens.Coming back to the topic, machine learning is implemented when:* A pattern exists* The pattern cannot be defined mathematically. At least no one knows that right now* The data must existLet’s take a look at the above points. For us to implement machine learning a pattern must exist. Machine learning cannot help if there is no pattern. If the distribution is truly random, then no amount of machine learning will be able to help. We can always fit a pattern in any set of data, but once new random data comes in, it will throw all our predictions to the dust. Another question may be how to understand if there is a pattern in the data from before. One of the greatest dichotomies of machine learning is that we don’t. We get a ‘feel’ of the data and based on that we apply the techniques and hope that it works. Sounds more like ‘magic’ and not at all the ‘science’ in ‘Data Science’.Now looking at the second point. The pattern should not be that obvious so that it can easily be defined mathematically. If it were, we could have just written a program and be done with it. Implementing machine learning models is ill-advised in such cases, as there is a certain amount of uncertainty in it. Implementation of such a model will never give us that kind of accuracy that we get by simply writing the code. So go along with machine learning only if the pattern cannot be identified exactly.Lastly, the data must exist. If there is no data, then there is nothing else to do. Period.The ProcessWe are going to describe a home as a vector with each component of the vector corresponding to some attribute of the home. Hence, the first field may be the area of the house. Second, the distance of the house from the nearest store and so on.[a1 -> area of the housea2 -> distance of the house from the nearest storea3 -> previous selling price of the house.…an]Such a house will have an optimum price that can be said to be the value of the house. We can say that if the above vector is called x, then there exists some function f, such that f(x) = y where y is the value of the house. Machine learning aims to find the value of f which is the ideal formula. The issue is that this function f is unknown and can be anything. This function f is the set of all functions that exists, which is infinite. So, how to find this function f . We say that this function f is uncomputable. Instead of using the data set that we have, we will formulate x and y and then we will try to guess, using machine learning methods, a function g such that g ≈ f (g is an approximation of f). So while f is unknown to us, g is very much known and is called the hypothesis set. We are hoping here that g is a good estimate of the function f .The Hypothesis Set and Learning AlgorithmWe can have the whole thing in the form of the following diagram.So, we start by first defining x and y and asking if there is a function f such that f(x) = y. Then we take the data set and come up with a learning algorithm where we generate a function g such that g is a good approximation of f. There is another crucial thing that is needed in the mix and it is the hypothesis set. The Learning algorithm and the Hypothesis set are the tools that we chose in order to come to the final solution. The Hypothesis set is the set of functions that have some common and unique properties that make them a viable candidate to be considered for the final hypothesis. We apply the learning algorithm to the hypothesis set and then choose a specific function from the hypothesis set. In terms of examples, the Hypothesis set could be linear regression and the learning algorithm could be gradient descent or the hypothesis set could be neural networks and the learning algorithm could be back propagation. Therefore, in order words, hypothesis set is a set of similar functions which have been shown to have good results under specific conditions and the learning algorithm is the algorithm that will do the actual searching. It will search through the space of possible hypotheses and produce the specific function that we will consider.An example of machine learningNow, working with the example of predicting home property values, let’s take a simple hypothesis set — linear regression. Let’s say take only one dimension of x, or one feature, which is the area of the house for simplicity.So, for input x = (x1, x2, x3, … xN) which are the corresponding areas of N houses.We will say that output y = (y1, y2, y3, …, yN) are the corresponding values of the houses. Say, this data set can be plotted in the graph shown below. Please note that this graph is possible because we are only taking one feature of the data set. In many cases, the data set would be of huge dimension and so we may not be able to visualize the graph but, would need to resort to other methods to get the feel of the data.Now, we will try to fit a linear line into the model. Since we have taken the hypothesis set to be linear regression the family of equations that we will consider is shown below.We can convert this equation in matrix form, which can be written in the following format:where h is the output vector, Θ and X are the model weights and the input matrices respectively. To start, we initialize Θ to random values. This gives us a random line.Now that we have the hypothesis set, we will decide on a learning algorithm. The learning algorithm tells us the resources we can work with. In this case, we will choose the gradient descent to be the learning algorithm. Now, we will run through the whole training set and form the values of h for each value of x . Applying the gradient descent algorithm, it updates the weight vector Θ such that next time the error of h from the real values of y is lessened. In this way, after iterating through the training set, hundreds of thousands of time we arrive at a line or at a value of Θ that represents the final hypothesis g.Machine learning is slowly making inroads into all the aspects of our lives and soon it would be imperative for us as developers to be able to understand data and generate models. I hope through this post you have been able to grasp some of the beauty that encompasses this interesting field.

Learn More >

Managing C# Dependencies: A Dive into Dependency Injection

Introduction:Have you ever struggled to manage C# dependencies in your projects? As your codebase grows, manually handling multiple dependencies can become daunting. Thankfully, there’s a solution that can streamline your code and improve its maintainability: dependency injection (DI). In this blog post, we’ll have a conversation where we explore a real-life scenario, discuss the challenges faced, and see how implementing dependency injection can provide an optimal solution.User: Hi there! I’ve heard much about dependency injection in C# lately, but I’m not sure how to implement it. Can you give me a beginner-friendly overview?C#: Absolutely! Dependency injection is a design pattern that helps manage the dependencies between components in your code. It allows you to write more modular and flexible code by reducing direct dependencies and promoting loose coupling.User: That sounds interesting! How can I start implementing dependency injection in my C# projects?C#: To implement dependency injection in C#, you can use a DI container or a DI framework. One popular choice is Microsoft.Extensions.DependencyInjection framework. It provides a lightweight and easy-to-use DI container. Let’s walk through an example to illustrate how it works.User: Great! I’m excited to see an example. Please show me how to use it.C#: Sure! First, you’ll need to install Microsoft.Extensions.DependencyInjection NuGet package. Once that’s done, you can create an instance of the ServiceCollection class, the DI container provided by the framework. Here’s a simple example:using Microsoft.Extensions.DependencyInjection;using System;// Define a class with a dependencypublic class Logger{ private readonly ILoggingService _loggingService; public Logger(ILoggingService loggingService) { _loggingService = loggingService; } public void Log(string message) { _loggingService.LogMessage(message); }}// Define an interface for the dependencypublic interface ILoggingService{ void LogMessage(string message);}// Implement the dependencypublic class ConsoleLogger : ILoggingService{ public void LogMessage(string message) { Console.WriteLine($"Logging message: {message}"); }}public class Program{ public static void Main() { // Configure the DI container var serviceProvider = new ServiceCollection() .AddTransient<ILoggingService, ConsoleLogger>() .BuildServiceProvider(); // Resolve the dependency var logger = serviceProvider.GetService<Logger>(); // Use the dependency logger.Log("Hello, world!"); }}In this example, we have a class called Logger that depends on an interface ILoggingService. The dependency is injected into Logger its constructor. By registering the dependency AddTransient in the DI container, we ensure that a new dependency instance is created each time it's requested.User: Ah, I see! So the DI container creates instances and injects them into the needed classes?C#: Exactly! The DI container manages the creation and injection of dependencies for you. It resolves the dependencies based on their registered types and provides them with the required classes.User: That’s convenient! Are there different ways to register dependencies in the DI container?C#: Absolutely! The ServiceCollection class provides various extension methods to register dependencies with other lifetimes. The most common ones are:AddTransient: A new dependency instance is created each time it’s requested.AddScoped: A new example of the dependency is created once per scope (e.g., per web request or logical operation).In conclusion, implementing dependency injection in our C# projects can greatly simplify managing dependencies and improve code maintainability. While Microsoft.Extensions.Dependency injection is a popular choice; we also have alternatives like Autofac, Ninject, Unity, and Simple Injector. Evaluate the features and characteristics of each framework to determine the best fit for your project. Happy coding!

Learn More >

Agile Methodology Unveiled

Agile Methodology, to me, seems like a distant cousin of division of labor!Wait, before you call it a technical term!The word agile has been so much in use in the technology industry that now when we think of if it, the term seems connotative of the domain. Even, the dictionary meaning has one general definition and one specific to software development.When diving into the origin of this word or methodology, you will find that it was not connected even remotely to the technological sphere. Time and again the innovative minds of people came up with methods which gave us a glimpse of this method. Be it the formation of Shewhart Cycle (Plan-do-Check-Act) or the implementation of this in the post WWII Japan by W. Edwards Deming.However, the concept took some time to take concrete shape. It was February 2001, when 17 exponents believing in various methodologies from the world of software development decided to bring in a methodology which all agreed on. Thus, in the meeting held at Snowbird, Utah, the term ‘agile’ was adopted for this methodology and the Agile Manifesto came into existence, and so did the core value:Individuals and interactions over processes and toolsWorking software over comprehensive documentationCustomer collaboration over contract negotiationResponding to change over following a planAs we can gauge, the first half of each value gains more importance over the second half, though both are considered in the development cycle.With the changing times, it was becoming difficult to carry on the waterfall and other traditional methodologies in product development. Agile came as an answer to these difficulties and has since then proved to be efficient.This iterative process allows us to make changes as and when needed rather than wait for the entire product to be developed before we get to know about any glitches.Remember the comparison between agile and division of labor?This is where scrum comes into picture. Out of the many agile practices like XP (eXtreme Programming), DSDM (Dynamic systems development method) and FDD (Feature-Driven Development), Scrum is the most popular one.All the requirements/features of a product are termed as user stories each and these stories sum up to a product backlog. This is further divided into the release backlog which is again divided into sprints (typically a sprint extends upto two weeks or more depending on the release cycles). At the end of a sprint, it is necessary to have a ship ready product and if there is any delay, it calls for change in plan or retrospection.This brings in the concept of a burndown chart. Through this chart we get to get a clear view of the progress of the project, especially the work which is undone. This chart also helps to estimate the sprint time as well as the time when the product will be ready for release.But, how do we keep a track of all these?The Scrum (stand-up) meetings… rings a bell?Yes, these meetings are integral to a sprint planning and every day all the members of the team meet for 15 minutes to discuss the progress of the project. These meetings are mostly about three questions:(1) What they have worked on since last scrum meeting(2) What they are planning to work on in that day &(3) If there are any problems they are facingTherefore, you now know the importance of these meetings.This was a glimpse into the world of agile. We will soon be back with other aspects of it.This then, be as agile as possible. Remember! It’s not a technical term and can be applied to any sphere of life.

Learn More >

Scope Chain Performance Optimization Tips

Browsers have evolved a lot in the past few years and many of the below discussed points may or may not be valid for modern browsers.However, I am just catching up with few points regarding scope chain and performance that may (or may not in modern browsers) affect the performance of web apps.Lets start with an example:While testScopeChain function is invoked, execution context sets up a stack. The local variables will be accessed first. If a variable is not found, interpreter will go deeper in the stack and keep looking till it finds the object (it is somewhat related to prototypal inheritance).function testScopeChain() {var location = document.location.href;var protocol = document.location.protocol;return (‘Location is: ‘ + location + ‘ & Protocol is: ‘ + protocol);}In the above code snippet, document is a global object. So the interpreter will look for the object in the first row of the stack first and won’t find it. It will have to traverse further to look for the object in the next row of execution stack (global object). This further traversing will take longer and may cause performance issues.The code may be refactored as:function testScopeChain() {var doc = document;var location = doc.location.href;var protocol = doc.location.protocol;return (‘Location is: ‘ + location + ‘ & Protocol is: ‘ + protocol);}Once we have a local copy of document in doc, it can be accessed locally (in the first row of the stack).Coming to closures, an extra row is added to the execution context stack.function outerScope() {var outerScopeVal = ‘Outer Scope’;function innerScope() {var innerScopeVal = ‘Inner Scope’;console.log(outerScopeVal);console.log(innerScopeVal);}innerScope();}outerScope();In the above code snippet, innerScopeVal can be accessed locally (in the first row of the execution context stack). But when it comes to the outer scope, outerScopeVal can be accessed from the next row which again may take some time.Issue with ‘with’ and ‘catch’ statements: They will add a row to the execution stack. Essentially this row will be the first one. So even local variables will be placed in the second row.Recommendations:1> Use local variables as much as possible.2> Avoid with and catch statements3> Minimize use of closures4> Always use var to prevent accidental declaration of global variable

Learn More >

A Day at Nineleaps, Where Work Meets Creativity

Hey! You forgot to login. Yes, I am talking to you.BeepYeah, better. Now you can go.Sometimes I wish they could hear what I say, but most of the times I am glad that they cannot :-)One by one they come and accommodate themselves in their places facing the motivation which adorns their desks. Some have written words to help them begin a new day with renewed zeal while others have miniatures of their dreams to help them do their best. The walls too echo inspiration with simple yet impacting quotes inscribed on them.As the day starts you see one person reminding the other about something, and then like relay-race you see the thought traveling in the floor, making all land in the conference room. Yes, I am talking about the standup meetings which are held every morning to get a gist of what each one has been doing and who will be working on which aspect during the day. Sorted!In a while, one can hear footsteps following the aroma of tea and coffee and when they are back with their cup full of rejuvenation, all I can hear is the rhythm of the keyboard; no not the musical instrument but the QWERTY. It has got a rhythm unique to each coder and trust me, you cannot find it anywhere else.Even without copyright, it is solely theirs, and with each sip of coffee, the sound increases exponentially.The silent breaks with the booting of the systems and thus, their day starts. With different projects running simultaneously, people need to brainstorm on each idea to make it work. Trello boards are created wherein everyone gives their inputs while taking note of the requirements posted by others, and thus work takes pace.There are times when an idea strikes one and that idea materializes with participation from all; be it the Hackathon, the Sports Day or a team outing.Once someone thought of a wish tree and within minutes it was reflected on the wall. From those wishes, you get to know how different people think and what matters to them. Although those are anonymous, you get to see the diversity present.Hope their wishes come true!Every team works in collaboration to bring in the best results and that is why coordination is the foremost requirement for any project.A new joinee is as much included as a senior employee in the assignments and this fosters better learning.While one screen has the wireframe, other has the matrix code showering on it and yet another has the work tested. In all the phases of creating an interface, you not only see the different teams working but also a play of all the emotions. Sometimes, you see them on the verge of giving up, while at times they are rejoicing a success. I have seen them disappointed, I have seen them celebrate and I have also seen them fall and rise back.When the lunch time comes, all move out heading towards the cafeteria. This is the time when you get to see a new avatar of most of the people. During this time, we hardly see them cracking codes; rather they prefer to crack jokes, and trust me, they can give any famous standup comedian a run for their money.Three words aptly define this time; chatter (talks), laughter (you know why) and clatter (the plates and spoons).When someone returns from their hometown, they bring the flavor of their land and everyone gets a mail inviting them to dive into the taste.After lunch, everyone’s pace gets a little slow and to get out of that mode, there is this riddle session wherein one will post a puzzle. You should see the enthusiasm with which each one awaits the question on Slack. Within minutes after the puzzle appears, multiple answers flow in, together with a flood of comments.As the day passes by, they again gear up to take the day’s work ahead and with their earphones plugged in, they are unaware of everything but their task. When the project calls for brainstorming, they come together, pitch in their ideas and draw a path to achieve the goal.It is not always work and work in Nineleaps, people here play with their creative bent of mind quite often. Once they wanted to fill a corner with liveliness. Next thing, paint splashed and overnight they wrote a story with colors.Often I see people pacing up and down the stairs to help one another. The best part about this place is that you need not hesitate to approach anyone if there is an issue. You never get a no for an answer when you seek guidance from anyone here.Sometimes too much thinking blocks our mind and we need a refreshment to get back on track. I see people going out to grab a cup of coffee from Café Coffee Day or treat themselves with a mouthwatering pizza at Domino’s, while all the tea lovers gather at Chai Point.Be that a foodie or a not so foodie person, Koramangala has a plethora to offer when it comes to food joints.Right from Indian, to Vietnamese, to Italian, you name it and you get it.Perks of having an office at one of the most happening places in Bangalore.Did I tell you about the ‘What’s On Your Mind’ thing? Oh! How could I forget that? So, there is a board on which people write their thoughts everyday and it helps them speak their minds out.It’s wonderful how only two three words can comprise a whole story.Time to go home, but there are many who do not like to break the flow and work till they reach what they set to achieve. Burning the midnight oil, they stay back, spending their nights in the comfort of the bean bags or sofa cum bed. To take a break, they sometimes hit the dart bull’s eye or score some crazy points in carrom.And here I stand to witness every detail of their day because it’s me through whom they enter and exit everyday.

Learn More >