python - 多線程中mysql插入出錯(cuò)
問題描述
在寫爬蟲中,我要把爬取到的數(shù)據(jù)存到數(shù)據(jù)庫(kù)中.每一個(gè)頁面里邊有很多條目,比如一個(gè)人的訪客可能有很多個(gè),于是插入卸載循環(huán)中,
try: sql_visitor=’INSERT INTO visitor (ownername,owneruid,visitorname,visitoruid,visittime) VALUE ('%s',%d,'%s',%d,'%s')’%(ownername,owneruid,visitorname,visitoruid,visitortime) print sql_visitor self.cursor.execute(sql_visitor) self.connect.commit()except Exception as e: print e
一個(gè)頁面一個(gè)線程,嫌棄慢的我開了5個(gè)
max_threads=5while uid < 8000000 or threadlist: for thread1 in threadlist:if not thread1.is_alive(): threadlist.remove(thread1) while len(threadlist) < max_threads and uid < 8000000:uid+=1thread2=threading.Thread(target=run,args=(uid,))thread2.setDaemon(True)thread2.start()threadlist.append(thread2) time.sleep(5)
運(yùn)行很順利:
INSERT INTO visitor (ownername,owneruid,visitorname,visitoruid,visittime) VALUE ('huosai7',4893,'Liang2017',7252799,'2017-5-22 21:06')INSERT INTO personalinfo (ownername,owneruid,jifen,huajiao,xiaomijiao,jinbi,haoyou,zhuti,rizhi,xiangce,fenxiang,kongjianfangwenliang,youxiangyanzheng,shipinrenzheng,juzhudi,chushengdi,shangcifabiaoshijian,shangcihuodongshijian,zuihoufangwen,zhuceshijian,zaixianshijian,shengri,xingbie) VALUE('huosai7',4893,0,0,0,0,0,0,0,0,0,0,0,0,'','','2100-01-01 12:00','2100-01-01 12:00','2100-01-01 12:00','2004-1-3 19:28',0,'2100-01-01 12:00',0)INSERT INTO visitor (ownername,owneruid,visitorname,visitoruid,visittime) VALUE ('龍樂',4894,'Liang2017',7252799,'2017-5-22 21:06')(1062, 'Duplicate entry ’4894-7252799-2017-05-22 21:06:00’ for key ’PRIMARY’')INSERT INTO personalinfo (ownername,owneruid,jifen,huajiao,xiaomijiao,jinbi,haoyou,zhuti,rizhi,xiangce,fenxiang,kongjianfangwenliang,youxiangyanzheng,shipinrenzheng,juzhudi,chushengdi,shangcifabiaoshijian,shangcihuodongshijian,zuihoufangwen,zhuceshijian,zaixianshijian,shengri,xingbie) VALUE('龍樂',4894,0,0,0,0,0,0,0,0,0,0,0,0,'','','2100-01-01 12:00','2100-01-01 12:00','2100-01-01 12:00','2004-1-3 20:21',0,'2100-01-01 12:00',0).......
于是我將max_thread設(shè)置成10,于是結(jié)果如下:
INSERT INTO visitor (ownername,owneruid,visitorname,visitoruid,visittime) VALUE ('xiao61',4889,'Liang2017',7252799,'2017-5-22 21:06')
(2006, ’MySQL server has gone away’)
INSERT INTO personalinfo (ownername,owneruid,jifen,huajiao,xiaomijiao,jinbi,haoyou,zhuti,rizhi,xiangce,fenxiang,kongjianfangwenliang,youxiangyanzheng,shipinrenzheng,juzhudi,chushengdi,shangcifabiaoshijian,shangcihuodongshijian,zuihoufangwen,zhuceshijian,zaixianshijian,shengri,xingbie) VALUE('xiao61',4889,0,0,0,0,0,0,0,0,0,0,0,0,'','','2100-01-01 12:00','2100-01-01 12:00','2100-01-01 12:00','2004-1-3 15:56',0,'2100-01-01 12:00',0)
(2006, ’MySQL server has gone away’)
INSERT INTO visitor (ownername,owneruid,visitorname,visitoruid,visittime) VALUE ('糊涂酷酷熊',4897,'Liang2017',7252799,'2017-5-22 21:06')
(2006, ’MySQL server has gone away’)
INSERT INTO personalinfo (ownername,owneruid,jifen,huajiao,xiaomijiao,jinbi,haoyou,zhuti,rizhi,xiangce,fenxiang,kongjianfangwenliang,youxiangyanzheng,shipinrenzheng,juzhudi,chushengdi,shangcifabiaoshijian,shangcihuodongshijian,zuihoufangwen,zhuceshijian,zaixianshijian,shengri,xingbie) VALUE('糊涂酷酷熊',4897,611,0,1655,0,0,2,0,0,0,34,0,0,'','','2007-3-27 00:37','2007-3-27 00:37','2007-3-27 00:37','2004-1-3 21:08',0,'2100-01-01 12:00',1)
(2006, ’MySQL server has gone away’).......
可以看出2006出來了,然后我將max_thread設(shè)置成30,然后結(jié)果如下:
就將,夠詳細(xì)嗎?不夠詳細(xì)還需要什么只管說!
問題解答
回答1:看這里,我猜你是用的是pymysql,它的線程安全描述為1,對(duì)應(yīng)的pep249里面做了詳細(xì)的描述:
Threads may share the module, but not connections.
線程可以共享模塊但不能共享連接。這也就是說你可能得在每個(gè)線程中創(chuàng)建一個(gè)連接。
吶~為什么不用orm來做呢?
相關(guān)文章:
1. node.js - mysql如何通過knex查詢今天和七天內(nèi)的匯總數(shù)據(jù)2. mysql 插入數(shù)值到特定的列一直失敗3. 360瀏覽器與IE瀏覽器有何區(qū)別???4. Python從URL中提取域名5. mysql - 百萬行的表中是否盡量避免使用update等sql語句?6. python - 在使用Pycharm時(shí)經(jīng)常看到如下的樣式,小括號(hào)里紅色的部分是什么意思呢?7. javascript - 新浪微博網(wǎng)頁版的字?jǐn)?shù)限制是怎么做的8. 怎么在網(wǎng)頁中設(shè)置圖片進(jìn)行左右滑動(dòng)9. javascript - 豆瓣的這個(gè)自適應(yīng)是怎么做的?10. javascript - 用jsonp抓取qq音樂總是說回調(diào)函數(shù)沒有定義
