Python development – scrap stream data

Looking for a Python scraping Developer in Boston MA area? Contact OT PROJECTS.


from lxml.html import fromstring
from lxml.etree import tostring

import urllib2


request=urllib2.urlopen("http://www.yahoo.com")



html=request.read()


try:

	root=fromstring(html)

except Exception, e:

	print "root", e
	continue
	
	
p_tags=root.xpath("//p[contains(@class,'stream-summary')]")




for p in  p_tags:
	try:
		print p.text
		print
	except Exception, e:
		print e
		exit()

print "End",



Looking for a Python scraping Developer in Boston MA area? Contact OT PROJECTS.

Python Developer – Send an email using Python

This example shows how to send an email (SMTP)
Full stack innovative Python Developer in Boston MA area Contact OT PROJECTS.
Make the needed changes where there’s *** in the code



import smtplib
import time
import sys

from email.mime.image import MIMEImage
from email.mime.multipart import MIMEMultipart
from email.mime.text import MIMEText
from email.header import Header
from email.utils import formataddr

print 'start'

#------------------------------------------------------
# change the parameters according to your SMTP settings
_my_name="*** Your name *** "
_smtp_domain='*** enter here your smtp server name ***'
_port=587 # put the right port
_login_user ='*** enter here the SMTP user name (usually your email) ****'
_password='*** enter here your password ***'
_from='%s <%s>' % (_my_name,_login_user)
_to='*** enter here the recipient email ***'
#------------------------------------------------------
html='

Hello,
This is an eample

' msg = MIMEText(html, 'html') msg['Subject'] = 'OT PROJECTS - a Python sample email procedure' msg['From'] = _from msg['To'] = _to msg['Reply-to']=_from tries=0 while tries<3: # for login try: email_server = smtplib.SMTP(_smtp_domain,_port) email_server.login(_login_user,_password) tries=4 # exit the try except: print 'email error, wait 10 sec.' time.sleep(4) tries+=1 # try again after 10 seconds if tries==3: sys.exit('Error login in to email server') try: rv=email_server.sendmail(_from, _to, msg.as_string()) except Exception as e: print 'Error sending email: ', e email_server.quit() print 'send email completed'

Python Developer in Boston MA area Contact OT PROJECTS.

Python Developer – Python socket example.

Socket server by default is TCP IP. Save the 2 codes in 2 files, run the server first.
* Note the client IP is pointed to 127.0.0.1, assuming the client and the server are on the same machine.

Python Backend Developer in Boston MA area Contact OT PROJECTS.

This is the server file code:



#!/usr/bin/python
import socket
import time

print "Socket server started"
socket_server = socket.socket()         
ip = '0.0.0.0' 
port = 5050 #choose any, larger than 1024 (otherwise you need to have admin/root permission to execute)               
socket_server.bind((ip, port))        

socket_server.listen(5)                 
while True:
   client, client_address = socket_server.accept()     
   print client_address, 'connected'
   client.send('Connection time: ' + str(time.asctime(time.localtime())))
   client.close()                


This is the client file code:



#!/usr/bin/python
import socket 
import time 
       
ip_of_the_socket_server = '127.0.0.1'
port_of_the_socket_server = 5050     # must be the server port number       

#send 10 messages, 10 sec delay between each
for i in range(1,11):
	my_connection = socket.socket()  
	my_connection.connect((ip_of_the_socket_server, port_of_the_socket_server))	
	print i,')', my_connection.recv(4096)
	my_connection.close
	time.sleep(10)




Python Developer in Boston MA area Contact OT PROJECTS.

SQL Developer – Creating a Stored Procedure – MSSQL (SQL server) and MySQL

Stored procedure is a code stored in the database and executed on the server. The value is in many aspects, but can summarised as a backend operation running on the DB server.

If you like to run the code, user command lines or install SQL client tools:

  • MSSQL – SQL Server Management Studio or Visual Studio
  • MySQL – There are a few tools like MySQL client and MySQL Workbench.

    a few differences between MSSQL and MySQL:

    • MySQL – note the DELIMITER
    • MySQL – remember to close operations with ; (semicolon)
    • Auto Number: IDENTITY (MSSQL) -> AUTO_INCREMENT (MySQL)
    • AUTO_INCREMENT (MySQL) must be in the PRIMARY KEY

    SQL Server

    * Run each piece of code in a separate query (if using SQL management studio)

    For preparations, create the DB, table, index and enter some sample data.

    
    
    -- step 1 - create a database
    CREATE DATABASE otp;
    
    
    
    
    
    
    -- step 2 - use the new DB
    USE otp;
    -- step 3 - create a table
    CREATE TABLE
    	otp_demo
    	(
    		id INT IDENTITY,
    		p_name VARCHAR(200),
    		p_city VARCHAR(200) DEFAULT 'Boston',
    		p_state	VARCHAR(100) DEFAULT 'MA'
    	
    	)
    ;
    -- step 4 - create an index
    CREATE INDEX
    	inx_otp_demo
    ON 
    	otp_demo
    	(
    		p_name,
    		p_city,
    		p_state
    	)
    ;
    INSERT INTO otp_demo
    	(p_name,
    	p_city,
    	p_state
    	)
    VALUES
    	('Dan','Brookline', 'MA'), 
    	('Dana','Boston', 'MA'),
    	('Carla','Newton', 'MA'),
    	('Deborah','Cambridge', 'MA')
    
    
    
    
    

    Create a simple stored procedure

    
    
    CREATE PROCEDURE otp_count_names
    AS
    	BEGIN
    		
    		SELECT 
    			COUNT(*)
    		FROM
    			otp_demo
    	
    	END	
    
    
    

    Run the stored procedure.

    
    
    EXEC otp_count_names
    
    
    

    Result: 4

    MySQL

    * Note there are small differences

    • IDENTITY -> AUTO_INCREMENT
    • AUTO_INCREMENT must be in the PRIMARY KEY

    For preparations, create the DB, table, index and enter some sample data.

    
    
    -- step 1 - create a database
    CREATE DATABASE otp
    
    
    
    
    
    
    USE otp;
    -- step 3 - create a table
    CREATE TABLE
    	otp_demo
    	(
    		id INT AUTO_INCREMENT,
    		p_name VARCHAR(200),
    		p_city VARCHAR(200) DEFAULT 'Boston',
    		p_state	VARCHAR(100) DEFAULT 'MA',
    	PRIMARY KEY (id)
    	
    	)
    ;
    -- step 4 - create an index
    CREATE INDEX
    	inx_otp_demo
    ON 
    	otp_demo
    	(
    		p_name,
    		p_city,
    		p_state
    	)
    ;
    INSERT INTO otp_demo
    	(p_name,
    	p_city,
    	p_state
    	)
    VALUES
    	('Dan','Brookline', 'MA'), 
    	('Dana','Boston', 'MA'),
    	('Carla','Newton', 'MA'),
    	('Deborah','Cambridge', 'MA')
    
    
    

    Create a simple stored procedure

    
    
    DELIMITER $$
    CREATE PROCEDURE otp_count_names()
    
    	BEGIN
    		
    		SELECT 
    			COUNT(*)
    		FROM
    			otp_demo;
    	
    	END $$	
    
    
    
    

    Run the stored procedure.

    
    
    CALL otp_count_names
    
    
    

    Result: 4

Python Developer – Python lists

Lists in Python are like Arrays, and are modifiable (in oppose to tuples).

Examples:

names=['Dan', 'Iris','Carla','Adam']
    The list length:
len(names)

Result:? 4

You can populate the list this way:


names=[]
names.append('Dan')
names.insert('Iris')
names+=['Carla','Adam']

    To print the names
?print names

Result: [‘Dan’, ‘Iris’, ‘Carla’, ‘Adam’]

    Print each name in names
for name in names:
	print name

Result:
Dan
Iris
Carla
Adam

You can do this as well:


for n in range(len(names)):
	print names[n]

Result:
Dan
Iris
Carla
Adam

Print the second value in names.
(Since sequence starts at index= 0, second is index is 1):


print names[1] 

Result: Iris

Print the values from the 2nd (index=1) to the 3th (you’ll need to assign index=3!)


print names[1:3]

Result: [‘Iris’, ‘Carla’]

Sort the list:


names= sorted(names)
print names

Result: [‘Adam’, ‘Carla’, ‘Dan’, ‘Iris’]

Check if value exists:


print 'Dan' in names

result: True
note: Python is case sensitive.


print 'dan' in names

result: False

Modify the second value:


names[2]='Gary'
print names

Result: [‘Adam’, ‘Dan’, ‘Gary’, ‘Iris’]

Add a new value (between existing values) at index=2 (3rd location):


names[2:2]=['Karen']
print names

Result: [‘Adam’, ‘Carla’, ‘Karen’, ‘Dan’, ‘Iris’]


names.insert(2,'Karen')
print names

Result: [‘Adam’, ‘Carla’, ‘Karen’, ‘Karen’, ‘Dan’, ‘Iris’]

Delete a value:


names.remove('Karen')
print names

Result: [‘Adam’, ‘Carla’, ‘Karen’, ‘Dan’, ‘Iris’]
* note just one value was deleted.

Delete a value by index:


del names[2)
print names

Result: [‘Adam’, ‘Carla’, ‘Dan’, ‘Iris’]

Count the appearances of a value:


print  names.count('Iris')


Result: 1

Working with multiple Lists:

instantiate


names=['Adam', 'Carla', 'Dan', 'Iris']
names1=names
names1+=['Deborah']
print names
print names1

Result:
[‘Adam’, ‘Carla’, ‘Dan’, ‘Iris’, ‘Deborah’]
[‘Adam’, ‘Carla’, ‘Dan’, ‘Iris’, ‘Deborah’]

Fork/copy


names=['Adam', 'Carla', 'Dan', 'Iris']
names1=list(names)
names1+=['Deborah']
print names
print names1

Result:
[‘Adam’, ‘Carla’, ‘Dan’, ‘Iris’]
[‘Adam’, ‘Carla’, ‘Dan’, ‘Iris’, ‘Deborah’]

another way by using the copy lib:


from copy import copy

names=['Adam', 'Carla', 'Dan', 'Iris']
names1=copy(names)
names1+=['Deborah']
print names
print names1

Result:
[‘Adam’, ‘Carla’, ‘Dan’, ‘Iris’]
[‘Adam’, ‘Carla’, ‘Dan’, ‘Iris’, ‘Deborah’]

Merge lists


names=['Adam', 'Carla', 'Dan', 'Iris']
names1=list(names)
names2=names+names1
print 'names:', names
print 'names1:',names1
print 'names2:',names2

Result:
names: [‘Adam’, ‘Carla’, ‘Dan’, ‘Iris’]
names1: [‘Adam’, ‘Carla’, ‘Dan’, ‘Iris’, ‘Gail’]
names2: [‘Adam’, ‘Carla’, ‘Dan’, ‘Iris’, ‘Adam’, ‘Carla’, ‘Dan’, ‘Iris’, ‘Gail’]

Full stack innovative Python Developer in Boston MA area Contact OT PROJECTS.