Python development – scrap stream data

Looking for a Python scraping Developer in Boston MA area? Contact OT PROJECTS.


from lxml.html import fromstring
from lxml.etree import tostring

import urllib2


request=urllib2.urlopen("http://www.yahoo.com")



html=request.read()


try:

	root=fromstring(html)

except Exception, e:

	print "root", e
	continue
	
	
p_tags=root.xpath("//p[contains(@class,'stream-summary')]")




for p in  p_tags:
	try:
		print p.text
		print
	except Exception, e:
		print e
		exit()

print "End",



Looking for a Python scraping Developer in Boston MA area? Contact OT PROJECTS.

Python Developer – Send an email using Python

This example shows how to send an email (SMTP)
Full stack innovative Python Developer in Boston MA area Contact OT PROJECTS.
Make the needed changes where there’s *** in the code



import smtplib
import time
import sys

from email.mime.image import MIMEImage
from email.mime.multipart import MIMEMultipart
from email.mime.text import MIMEText
from email.header import Header
from email.utils import formataddr

print 'start'

#------------------------------------------------------
# change the parameters according to your SMTP settings
_my_name="*** Your name *** "
_smtp_domain='*** enter here your smtp server name ***'
_port=587 # put the right port
_login_user ='*** enter here the SMTP user name (usually your email) ****'
_password='*** enter here your password ***'
_from='%s <%s>' % (_my_name,_login_user)
_to='*** enter here the recipient email ***'
#------------------------------------------------------
html='

Hello,
This is an eample

' msg = MIMEText(html, 'html') msg['Subject'] = 'OT PROJECTS - a Python sample email procedure' msg['From'] = _from msg['To'] = _to msg['Reply-to']=_from tries=0 while tries<3: # for login try: email_server = smtplib.SMTP(_smtp_domain,_port) email_server.login(_login_user,_password) tries=4 # exit the try except: print 'email error, wait 10 sec.' time.sleep(4) tries+=1 # try again after 10 seconds if tries==3: sys.exit('Error login in to email server') try: rv=email_server.sendmail(_from, _to, msg.as_string()) except Exception as e: print 'Error sending email: ', e email_server.quit() print 'send email completed'

Python Developer in Boston MA area Contact OT PROJECTS.

Python Developer – Python socket example.

Socket server by default is TCP IP. Save the 2 codes in 2 files, run the server first.
* Note the client IP is pointed to 127.0.0.1, assuming the client and the server are on the same machine.

Python Backend Developer in Boston MA area Contact OT PROJECTS.

This is the server file code:



#!/usr/bin/python
import socket
import time

print "Socket server started"
socket_server = socket.socket()         
ip = '0.0.0.0' 
port = 5050 #choose any, larger than 1024 (otherwise you need to have admin/root permission to execute)               
socket_server.bind((ip, port))        

socket_server.listen(5)                 
while True:
   client, client_address = socket_server.accept()     
   print client_address, 'connected'
   client.send('Connection time: ' + str(time.asctime(time.localtime())))
   client.close()                


This is the client file code:



#!/usr/bin/python
import socket 
import time 
       
ip_of_the_socket_server = '127.0.0.1'
port_of_the_socket_server = 5050     # must be the server port number       

#send 10 messages, 10 sec delay between each
for i in range(1,11):
	my_connection = socket.socket()  
	my_connection.connect((ip_of_the_socket_server, port_of_the_socket_server))	
	print i,')', my_connection.recv(4096)
	my_connection.close
	time.sleep(10)




Python Developer in Boston MA area Contact OT PROJECTS.

SQL Developer – Creating a Stored Procedure – MSSQL (SQL server) and MySQL

Stored procedure is a code stored in the database and executed on the server. The value is in many aspects, but can summarised as a backend operation running on the DB server.

If you like to run the code, user command lines or install SQL client tools:

  • MSSQL – SQL Server Management Studio or Visual Studio
  • MySQL – There are a few tools like MySQL client and MySQL Workbench.

    a few differences between MSSQL and MySQL:

    • MySQL – note the DELIMITER
    • MySQL – remember to close operations with ; (semicolon)
    • Auto Number: IDENTITY (MSSQL) -> AUTO_INCREMENT (MySQL)
    • AUTO_INCREMENT (MySQL) must be in the PRIMARY KEY

    SQL Server

    * Run each piece of code in a separate query (if using SQL management studio)

    For preparations, create the DB, table, index and enter some sample data.

    
    
    -- step 1 - create a database
    CREATE DATABASE otp;
    
    
    
    
    
    
    -- step 2 - use the new DB
    USE otp;
    -- step 3 - create a table
    CREATE TABLE
    	otp_demo
    	(
    		id INT IDENTITY,
    		p_name VARCHAR(200),
    		p_city VARCHAR(200) DEFAULT 'Boston',
    		p_state	VARCHAR(100) DEFAULT 'MA'
    	
    	)
    ;
    -- step 4 - create an index
    CREATE INDEX
    	inx_otp_demo
    ON 
    	otp_demo
    	(
    		p_name,
    		p_city,
    		p_state
    	)
    ;
    INSERT INTO otp_demo
    	(p_name,
    	p_city,
    	p_state
    	)
    VALUES
    	('Dan','Brookline', 'MA'), 
    	('Dana','Boston', 'MA'),
    	('Carla','Newton', 'MA'),
    	('Deborah','Cambridge', 'MA')
    
    
    
    
    

    Create a simple stored procedure

    
    
    CREATE PROCEDURE otp_count_names
    AS
    	BEGIN
    		
    		SELECT 
    			COUNT(*)
    		FROM
    			otp_demo
    	
    	END	
    
    
    

    Run the stored procedure.

    
    
    EXEC otp_count_names
    
    
    

    Result: 4

    MySQL

    * Note there are small differences

    • IDENTITY -> AUTO_INCREMENT
    • AUTO_INCREMENT must be in the PRIMARY KEY

    For preparations, create the DB, table, index and enter some sample data.

    
    
    -- step 1 - create a database
    CREATE DATABASE otp
    
    
    
    
    
    
    USE otp;
    -- step 3 - create a table
    CREATE TABLE
    	otp_demo
    	(
    		id INT AUTO_INCREMENT,
    		p_name VARCHAR(200),
    		p_city VARCHAR(200) DEFAULT 'Boston',
    		p_state	VARCHAR(100) DEFAULT 'MA',
    	PRIMARY KEY (id)
    	
    	)
    ;
    -- step 4 - create an index
    CREATE INDEX
    	inx_otp_demo
    ON 
    	otp_demo
    	(
    		p_name,
    		p_city,
    		p_state
    	)
    ;
    INSERT INTO otp_demo
    	(p_name,
    	p_city,
    	p_state
    	)
    VALUES
    	('Dan','Brookline', 'MA'), 
    	('Dana','Boston', 'MA'),
    	('Carla','Newton', 'MA'),
    	('Deborah','Cambridge', 'MA')
    
    
    

    Create a simple stored procedure

    
    
    DELIMITER $$
    CREATE PROCEDURE otp_count_names()
    
    	BEGIN
    		
    		SELECT 
    			COUNT(*)
    		FROM
    			otp_demo;
    	
    	END $$	
    
    
    
    

    Run the stored procedure.

    
    
    CALL otp_count_names
    
    
    

    Result: 4

Python Programmer – Python Decorators

A simple example to illustrate the use of function decorator in Python.

  • _func – you call a function that is executed.

With methods.



print 'started'		
def _exp(_func):

	def do_exp(v):
		s=0			
		for i in v:
			s+=(i)
		m_exp=s**2
		
		return {'exp':m_exp,'sum':s}
	return do_exp

	

@_exp
def show_exp():
	return

	

	
print show_exp([2,2,3])
print show_exp([1,2,3])
print show_exp([2,15,3,4,5,6,7,8,9,1,3])


With a class.


class _math(object):
	def __init__(self):
		print 'started'		
	def _exp(self,_func):
	
		def do_exp(v):
			s=0			
			for i in v:
				s+=(i)
			m_exp=s**2
			
			return {'exp':m_exp,'sum':s}
		return do_exp

	
m=_math()
@m._exp
def show_exp():
	return

	

	
print show_exp([2,2,3])
print show_exp([1,2,3])
print show_exp([2,15,3,4,5,6,7,8,9,1,3])

Full stack innovative Python Developer in Boston MA area Contact OT PROJECTS.

Python Developer – Python lists

Lists in Python are like Arrays, and are modifiable (in oppose to tuples).

Examples:

names=['Dan', 'Iris','Carla','Adam']
    The list length:
len(names)

Result:? 4

You can populate the list this way:


names=[]
names.append('Dan')
names.insert('Iris')
names+=['Carla','Adam']

    To print the names
?print names

Result: [‘Dan’, ‘Iris’, ‘Carla’, ‘Adam’]

    Print each name in names
for name in names:
	print name

Result:
Dan
Iris
Carla
Adam

You can do this as well:


for n in range(len(names)):
	print names[n]

Result:
Dan
Iris
Carla
Adam

Print the second value in names.
(Since sequence starts at index= 0, second is index is 1):


print names[1] 

Result: Iris

Print the values from the 2nd (index=1) to the 3th (you’ll need to assign index=3!)


print names[1:3]

Result: [‘Iris’, ‘Carla’]

Sort the list:


names= sorted(names)
print names

Result: [‘Adam’, ‘Carla’, ‘Dan’, ‘Iris’]

Check if value exists:


print 'Dan' in names

result: True
note: Python is case sensitive.


print 'dan' in names

result: False

Modify the second value:


names[2]='Gary'
print names

Result: [‘Adam’, ‘Dan’, ‘Gary’, ‘Iris’]

Add a new value (between existing values) at index=2 (3rd location):


names[2:2]=['Karen']
print names

Result: [‘Adam’, ‘Carla’, ‘Karen’, ‘Dan’, ‘Iris’]


names.insert(2,'Karen')
print names

Result: [‘Adam’, ‘Carla’, ‘Karen’, ‘Karen’, ‘Dan’, ‘Iris’]

Delete a value:


names.remove('Karen')
print names

Result: [‘Adam’, ‘Carla’, ‘Karen’, ‘Dan’, ‘Iris’]
* note just one value was deleted.

Delete a value by index:


del names[2)
print names

Result: [‘Adam’, ‘Carla’, ‘Dan’, ‘Iris’]

Count the appearances of a value:


print  names.count('Iris')


Result: 1

Working with multiple Lists:

instantiate


names=['Adam', 'Carla', 'Dan', 'Iris']
names1=names
names1+=['Deborah']
print names
print names1

Result:
[‘Adam’, ‘Carla’, ‘Dan’, ‘Iris’, ‘Deborah’]
[‘Adam’, ‘Carla’, ‘Dan’, ‘Iris’, ‘Deborah’]

Fork/copy


names=['Adam', 'Carla', 'Dan', 'Iris']
names1=list(names)
names1+=['Deborah']
print names
print names1

Result:
[‘Adam’, ‘Carla’, ‘Dan’, ‘Iris’]
[‘Adam’, ‘Carla’, ‘Dan’, ‘Iris’, ‘Deborah’]

another way by using the copy lib:


from copy import copy

names=['Adam', 'Carla', 'Dan', 'Iris']
names1=copy(names)
names1+=['Deborah']
print names
print names1

Result:
[‘Adam’, ‘Carla’, ‘Dan’, ‘Iris’]
[‘Adam’, ‘Carla’, ‘Dan’, ‘Iris’, ‘Deborah’]

Merge lists


names=['Adam', 'Carla', 'Dan', 'Iris']
names1=list(names)
names2=names+names1
print 'names:', names
print 'names1:',names1
print 'names2:',names2

Result:
names: [‘Adam’, ‘Carla’, ‘Dan’, ‘Iris’]
names1: [‘Adam’, ‘Carla’, ‘Dan’, ‘Iris’, ‘Gail’]
names2: [‘Adam’, ‘Carla’, ‘Dan’, ‘Iris’, ‘Adam’, ‘Carla’, ‘Dan’, ‘Iris’, ‘Gail’]

Full stack innovative Python Developer in Boston MA area Contact OT PROJECTS.

Python Programmer – Write and check JSON file with Python




import json
import pprint

#generate the JSON file
#will have 2 fields, one is an array (multiple records inside)

#*** make sure the fields and data are written in double quotes

my_json={}
my_json["description"]="This is a json file creation example written in Python"
my_json["data"]=[]
my_data_fields=['id','name']
for i in range(10):	
	my_json["data"].append({"id":i,"name":"name "+str(i)})

#take a look at the JSON output
print my_json

#write the file to the disk
fn='otp.json'
f=open(fn,"w")
f.write(json.dumps(my_json))
f.close()

#Now check if the JSON is valid (can be read as JSON)
json_data=open(fn)

data = json.load(json_data)
pprint.pprint(data)
json_data.close()

Python Developer- insert data records to a database (SQL)

Here is an example for inserting data to SQLite database.
You can see how to create a table, check if a table exists, and the use of single and multiple record insertion.


import sqlite3
import os



#-----------------------------------------------------------------------
#you need the path for the database
script_directory=os.getcwd()+os.path.sep

print script_directory

#this is the db we're creating

file_name="pydb.sqlite"
table_name='xx'
#Connect to the database
# Note - if an SQLite file desn't exists, the connect method will create it. If it exists, it'll open it.
con=sqlite3.connect(file_name)
#Open a cursor-this will move through a record set and read (or write)
cur=con.cursor()

#-----------------------------------------------------------------------

#Create a table, if not exist

sql=""" SELECT COUNT(*) FROM sqlite_master WHERE type='table' AND tbl_name=? """
cur.execute(sql,[table_name])
table_exits=cur.fetchone()[0]
print table_exits
if table_exits==0:
	print 'create the %s table '% table_name
	# This code can run with any DB, not only SQLite
	sql=""" 
		CREATE TABLE %s
			(
				id INT,
				name VARCHAR(200)
			
			)
		""" % table_name
	#run the creation with the connect object.
	con.execute(sql)

#If you need a GUI for SQLite, one option is to use the FireFox add-on, works great.

#-----------------------------------------------------------------------
#-----------------------------------------------------------------------

#Insert statement for inserting new values
#it's alway good practice to spell the field names


sql="""INSERT INTO %s (id,name)""" % table_name


#The VALUES are 2 parameters that will follow later when executing
# the ? is a parameter

# Note - this may NOT work the same with different databases.

sql+=""" VALUES (?,?) """

#Here are 2 ways to add a record:
#-----------------------------------------------------------------------
#1) record by record 

#the parameters are a list

#inserting 2 records one by one
for i in range(1,3):
	
	params=[i,'Person '+str(i)] #'Person '+str(i) will make the names unique
	#execute via cursor 
	cur.execute(sql,params)
	# to save the new record, use commit method of the connection.
	con.commit()
	print 'run no. ',i
#-----------------------------------------------------------------------
#2) insert multiple (2) records - faster and efficient

#the parameters are lists in a list

params=[] # Note - you assign/declare the main list here
for i in range(3,5):
	params.append([i,'Person '+str(i)]) #append a list for each record
	
#execute via cursor 
cur.executemany(sql,params) # run all at once
# use commit method of the connection.
con.commit()
print '1 run  for all'
#-----------------------------------------------------------------------

Boston SQL developer MySQL – write from a table to a text file

Write an output file from MySQL:



use test;
DELIMITER $$
DROP PROCEDURE IF EXISTS write_output_file;
CREATE PROCEDURE write_output_file(flag varchar(1))
       BEGIN
	DECLARE s  VARCHAR(255);
	SET s='this is an example';	
	IF flag	='r' THEN 
		-- Note you can't overwrite an existing file
		SELECT s INTO OUTFILE '/home/USER/output.txt'; -- write to a file
	ELSE
		SELECT s; -- print to the console
		
	END IF;
       END$$		
   DELIMITER ;

-- Run this procedure
call write_output_file('r')


Boston Django developer – Setting up SSL (nginx+Django)

Start with generating the SSL (here is a self-sign for development only):

Independent Django Developer in Boston MA area Contact OT PROJECTS.

Note: I use /home/user/ssl/ for the certification location – change accordingly




Topic:	SSL - django
Description:	create SSL #1
Sample:	openssl genrsa -des3 -out server.key 1024
------------------------------------------------------

Topic:	SSL - django
Description:	create SSL #1.1
Sample:	enter the pass: testssl
------------------------------------------------------

Topic:	SSL - django
Description:	Generate a CSR (Certificate Signing Request)  #2
Sample:	openssl req -new -key server.key -out server.csr
------------------------------------------------------

Topic:	SSL - django
Description:	Generate a CSR (Certificate Signing Request)  #2.1
Sample:	pass: testssl
------------------------------------------------------

Topic:	SSL - django
Description:	Generate a CSR (Certificate Signing Request)  #2.2
Sample:	Country: US
------------------------------------------------------

Topic:	SSL - django
Continue with writing accordingly:
Description:	Generate a CSR (Certificate Signing Request)  #2.3
Sample:	Country Name (2 letter code) [AU]:US
State or Province Name (full name) [Some-State]:massachusetts
Locality Name (eg, city) []:boston
Organization Name (eg, company) [Internet Widgits Pty Ltd]:OT PROJECTS
Organizational Unit Name (eg, section) []:development
Common Name (e.g. server FQDN or YOUR name) []:testssl
Email Address []:f@testssl.com

Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:testssl
An optional company name []:testssl

------------------------------------------------------

Topic:	SSL
Description:	Remove Passphrase from Key #2.4
Sample:	        cp server.key server.key.org
                openssl rsa -in server.key.org -out server.key
------------------------------------------------------

Topic:	SSL
Description:	Generating a Self-Signed Certificate  #2.5
Sample:	        openssl x509 -req -days 365 -in server.csr -signkey server.key -out server.crt
------------------------------------------------------

Topic:	SSL
Description:	Installing the Private Key and Certificate  #2.6
Sample:	        cp server.crt /home/user/ssl/conf/server.crt
                cp server.key /home/user/ssl/server.key
------------------------------------------------------



Now setup Nginx conf file:
if you use ubuntu: place a custom conf file here: /etc/nginx/conf.d
e.g.: /etc/nginx/conf.d/otp.conf:



server {
    listen          80;
    server_name     localhost;
    rewrite ^/(.*)  https://localhost/$1 permanent;
}

server {
    listen          4443 ssl;
    server_name     localhost;
    #rewrite ^/(.*)  https://localhost:4443/$1 permanent;
    access_log      /home/user/env/access.log combined;
    error_log       /home/user/env/error.log error;

    ssl_certificate         /home/user/ssl/server.crt;
    ssl_certificate_key     /home/user/ssl/server.key;

    location /static/ {
        alias /static/;
    }

    location /media/ {
        alias /media/;
    }

    location / {
        proxy_pass         http://localhost:8000/;
        proxy_redirect     off;

        proxy_set_header   Host              $http_host;
        proxy_set_header   X-Real-IP         $remote_addr;
        proxy_set_header   X-Forwarded-For   $proxy_add_x_forwarded_for;
    }

}






Independent Django Developer in Boston MA area Contact OT PROJECTS.

Restart nginx



 sudo /etc/init.d/nginx restart



Add to settings.py




SESSION_COOKIE_SECURE = True
CSRF_COOKIE_SECURE = True
SESSION_EXPIRE_AT_BROWSER_CLOSE=True




Now instead of localhost:8000 use localhost:4443

Finally, add this to your wsgi.py




os.environ['wsgi.url_scheme'] = 'https'



Should be the same, unless you used HTTP in your code/HTML or JS. Change accordingly.

Independent Django Developer in Boston MA area Contact OT PROJECTS.