How to install scala and create a class on Win & Linux

1 – Verify the JDK installation on your machine. Open the shell/terminal and type java -version and javac -version.

2 – Download Scala Binaries from http://www.scala-lang.org/download/. As of writing this post Scala version is 2.11.6, so you should be getting downloaded file as scala-2.11.6.tgz. Unzip the scala-2.11.6.tgz file using the following command as shown below.

3 – tar -xvzf scala-2.11.6.tgz

4 – After unzipping, change the path to point to the directory using cd command as shown below.

5 – For instance my directory is Downloads in which Scala binaries are unzipped.

6 – Now we are in the downloads directory where Scala binaries are present. Just go to the bin directory.

7 – cd scala-2.11.6 / cd bin

8 – This is the Scala REPL shell in which we can type programs and see the outcome right in the shell.

Scala Hello World Example

class Student() {
var id:Int = 0
var age:Int = 0
def studentDetails(i:Int,a:Int) {
id = i
age = a
println(“Student Id is :”+id);
println(“Student Age is :”+age);
}
}

Output: defined class Student

Here we create a Student class and print the student details in the studentDetails method by passing student id and age as parameter. If there are no errors in the code then a message “defined class Student” is displayed.

Create the student object and invoke the studdetails method by passing the student id and age.

object Stud {
def main(args:Array[String]) {
val stu = new Student();
stu.studentDetails(10,8);
}
}

Returns: defined object Stud

How to get data from Twitch-API with python

Hello Everyone,

This is a small quick py script for how to call API and how to parse json to csv with pandas for the beginners.

1 – Install below libraries,

import pandas as pd
import requests
import json

2 – Set your static values, btw you can use those values with .yml file

url = “https://wind-bow.glitch.me/twitch-api/channels/”
# List of channels we want to access
channels = [“ESL_SC2”, “OgamingSC2”, “cretetion”, “freecodecamp”, “storbeck”, “habathcx”, “RobotCaleb”, “noobs2ninjas”,
“ninja”, “shroud”, “Dakotaz”, “esltv_cs”, “pokimane”, “tsm_bjergsen”, “boxbox”, “a_seagull”,
“kinggothalion”, “jahrein”, “thenadeshot”, “sivhd”, “kingrichard”]

file_name = “talih.csv”
location =”C:\\Users\\talih\\Desktop\\TwitchPy\\TwitchAPI\\”
“”” Those values can be used with .yml file “””

3 – Set your class,functions:
class apicrawler:

def __init__(self,url,channels,file_name,location):
self.url = url
self.channels = channels
self.file_name = file_name
self.location = location

def selectedchannelcrawler(url,channels,location,file_name):
channels_list = []
for channel in channels:
JSONContent = requests.get(url + channel).json()
channels_list.append([JSONContent[‘_id’], JSONContent[‘display_name’], JSONContent[‘status’],
JSONContent[‘followers’], JSONContent[‘views’]])

dataset = pd.DataFrame(channels_list)
dataset.columns = [‘Id’, ‘Name’, ‘Status’, ‘Followers’, ‘Views’]
dataset.dropna(axis = 0, how = ‘any’, inplace = True)
dataset.index = pd.RangeIndex(len(dataset.index))
dataset.to_csv(location + file_name, sep=’,’, encoding=’utf-8′)

4 – Call that class for your own values:
apicrawler.selectedchannelcrawler(url,channels,location,file_name)

Enjoy

How to use dbt in python environment

Dbt is usefull library for dwh to create a datamart or datamarts. You can find all details in dbt official pages.

I used a few times, so i can clarify for you how you can create a dbt models and dbt configs in your own project, you can do that like below step by steps;

1 – Create a profiles.yml file for DBT Profile. Specify your db connection information etc.
2 – Create a data_model folder like project_dir
3 – Create a .yml file for main project .yml file and you will call it like project_file
4 – Create your own dbt_runner file like dbt_runner.py and set it your execution configs
5 – Create a model folder, you will put your models in that folder
6 – Create a schema or model for yourself and put into that folder a xxxx.schema.yml file
6.1 – Put some table value like below;
bietl_patch:
constraints:
unique:
– somthng_id
not_null:
– somthng_id
– xxx_id

In the end, you will have like below folder and schema;
# DBT Profile. Specify your DB connection information etc.
profiles.yml on the root directory
bietl_data_model folder
bietl_datamarts.yml file
dbt_runner.py python file

bietl_data_model
> models
> specification of your models bietl
> bietl_datamarts.schema.yml
> sql files for using.sql

I’m executing that dbt in airflow das but I didn’t mention it, maybe in next post.