分類
Linux 程序

Django簡易搭建上傳文件

這裡使用Django+Gunicorn+Nginx的方式簡單運行一個小型webserver,實現一個簡單的上傳文件到服務器的功能(並不生成下載鏈接)。

啟動虛擬環境,安裝django和gunicorn:

pip install Django==2.0
pip install gunicorn
#進入要放置代碼的目錄並新建項目
django-admin startproject mysite
##或者在當前目錄建立項目
#django-admin startproject mysite .
cd mysite
#新建app
python manage.py startapp polls

先建立一個表格:

#polls/forms.py
from django import forms

class UploadFileForm(forms.Form):
    title = forms.CharField(label='密碼',max_length=20,widget=forms.PasswordInput)
    file = forms.FileField(label='文件',)

修改view:

#polls/views.py
import os
import subprocess
from django.core.files.storage import FileSystemStorage
from django.conf import settings
from django.shortcuts import render
from django.http import HttpResponse
from .forms import UploadFileForm

#handle file example with file
def handle_uploaded_file(f):
    with open('/file/should/be/saved/here/target.odt', 'wb+') as destination:
        for chunk in f.chunks():
            destination.write(chunk)

#another handle file example with filename
def handle_uploaded_file2(filename):
    msg=''
    try:
        targetZipFilePath = os.path.join(settings.BASE_DIR, filename)
        cmd1=subprocess.check_call(["unzip", "-o", targetZipFilePath, "-d", "/home/fred/workspace/"])
        if cmd1==0 :
            cmd2=subprocess.check_call(["cp", "-Rf", "/home/fred/workspace/dist", "/home/fred/"])
            if cmd2==0 :
                msg="deployed successfully"
            else:
                msg="error 2"
        else:
            msg="error 1"
    except:
        msg = 'error 0'
    return msg

def upload_file(request):
    if request.method == 'POST':
        form = UploadFileForm(request.POST, request.FILES)
        if form.is_valid() and request.POST['title']=='Secret':
#            handle_uploaded_file(request.FILES['file'])
#            return HttpResponse("上傳成功")
            myfile = request.FILES['file']
            fs = FileSystemStorage()
            filename = fs.save(myfile.name, myfile)
            uploaded_file_url = fs.url(filename)
            print(uploaded_file_url)
            res = handle_uploaded_file2(uploaded_file_url)
            return HttpResponse(res)
    else:
        form = UploadFileForm()
    return render(request, 'upload.html', {'form': form})

新建一個表格的模板:

#polls/templates/upload.html
<form enctype="multipart/form-data" action="/polls/upload/" method="post">
    {% csrf_token %}
    {{ form }}
    <input type="submit" value="上傳" />
</form>

新建一個url路由表:

#polls/urls.py
from django.urls import path

from . import views

urlpatterns = [
    path('upload/', views.upload_file, name='upload_file'),
]

修改項目路由:

#mysite/urls.py
from django.contrib import admin
from django.urls import include,path

urlpatterns = [
    path('polls/', include('polls.urls')),
    path('admin/', admin.site.urls),
]

修改項目設置:

#mysite/settings.py
INSTALLED_APPS = [    
    'polls.apps.PollsConfig',
#    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
#    'django.contrib.messages',
#    'django.contrib.staticfiles',
]
LANGUAGE_CODE = 'zh-Hant'
TIME_ZONE = 'Asia/Taipei'

然後在項目目錄(最上層)運行gunicorn就可以訪問了:

gunicorn mysite.wsgi --bind 127.0.0.1:3040

nginx中增加如下server即可在外網訪問了(鏈接應該是http://YourPublicIP:8081/polls/upload/):

server {
        listen 8081;
        server_name 127.0.0.1;
        charset utf-8;
        keepalive_timeout 60s;
        #access_log logs/django2a.access.log combined if=$loggable;
        
        location / {
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_redirect off;
    
            if (!-f $request_filename) {
                proxy_pass http://127.0.0.1:3040;
                break;
            }
    }
}

CentOS6+Django2+MySql

yum install MySQL-python
pip install mysqlclient
#如果import MySQLdb提示無libmysqlclient.so.18
#則建立軟連接如下
ln -s /usr/local/mysql/lib/libmysqlclient.so.18 /usr/lib64/libmysqlclient.so.18
分類
网站

使用 UpUp 讓網站離線可用

UpUp 是個很小(2.5 KB)的 Javascript 庫,但卻可以方便的實現網站的離線訪問。我之前有一個小項目,PHP 的後端和頁面,另外藉助 Apeche Cordova 用 HTML 實現了安卓的客戶端以供離線使用。最近用 Django 重新寫了,藉助 UpUp 的離線頁面,我不用爲了離線使用而再去生成一個安卓應用。

一般的網站,要使用 UpUp 是非常簡單的,只需要在網站的頂層文件夾引入對應的庫即可,但是 Django 的靜態文件一般是放在 static 下的,要想在項目根目錄下提供它們,可以這樣設置。

#項目的 url.py 文件
from django.urls import path
...
from . import views

urlpatterns = [
    ...
    path('upup.min.js', cache_page(60 * 60 * 48)(TemplateView.as_view(template_name="site/upup.min.js", 
  content_type='application/javascript', )), name='upup.min.js'),
    path('upup.sw.min.js', cache_page(60 * 60 * 48)(TemplateView.as_view(template_name="site/upup.sw.min.js", 
  content_type='application/javascript', )), name='upup.sw.min.js'),
]

然後將 upup.min.js 和 upup.sw.min.js 放在項目的模板目錄裏,比如 mysite/site/templates/site/ 中。

最後在 HTML 頁面中引用並設置需要離線的文件即可。

...
<script type="text/javascript" src="{% url 'upup.min.js' %}"></script>
<script type="text/javascript" src="{% static 'site/js/jquery-3.6.0.min.js' %}"></script>
<script>
UpUp.start({
  'content-url': '/site/',
  'assets': ['/static/site/js/jquery-3.6.0.min.js',
   '/static/site/js/site.js', '/static/site/css/site.css']

});
</script>

離線站點調試的過程中遇到的另外一個問題是,離線站點必須是 HTTPS 類型的加密頁面,但是本地配置 HTTPS 又略嫌繁瑣,其實只要再 Chrome 裏添加例外(chrome://flags/#enable-site-per-process)即可。火狐我還不知道要怎麼添加,最近安卓火狐的一系列更新都不盡人意,實在是有些令人擔憂。

分類
說說

210521

非常遺憾,本站似乎無法從國內直接打開了。

#天威寬帶
$ curl https://ft.shaman.eu.org/
curl: (28) Failed to connect to ft.shaman.eu.org port 443: Connection timed out

#移動網絡(2G 和 4G)
$ curl https://ft.shaman.eu.org/
curl: (7) Couldn't connect to server

通過測速網站的結果可以看到,國內的 DNS 服務可以正確解析出本站的 Cloudflare IP,但是無法打開頁面。國外則正常。

分類
网站

使用 Plausible 統計用戶事件

中文繁簡轉換工具用上 Bootstrap 後漂亮多了,使用 JavaScript 來請求表格減少了頁面的跳轉,用起來也比較方便。於是產生一個新需求就是想統計下「轉換」按鈕被點擊了多少次。關於自定義事件目標的具體介紹,可以參考 Plausible 的官方文檔Custom event goals,我這裏只是羅列下具體步驟。

前端 html 中的修改

確保 Plausible 配置有下面的第二行

<script async defer data-domain="<yourdomain.com>" src="https://plausible.yourdomain.com/js/plausible.js"></script>
<script>window.plausible = window.plausible || function() { (window.plausible.q = window.plausible.q || []).push(arguments) }</script>

在需要統計的地方執行這行 JavaScript

plausible('my_event');

Plausible 後臺的設置

打開 Plausible 後臺,在左上角的域名處選擇 Site Settings。在 Goals 區域點擊 + Add goal。然後 Goal trigger 選擇 Custom event,Event name 要輸入 JavaScritp 請求的參數,這裏就是「my_event」。最後點擊 Add goal 就完成了。

分類
网站

使用 datatables 展示表格

計算得出的中國城市飲用水中全氟辛烷磺酸(PFASs)的平均濃度(ng/L)

調查城市PFOAPFOS平均值18年人口 (万)所在省份
Anshan0.40.91.3341.8Liaoning
Atushi0.050.050.128.55Xinjiang
Baoding0.560.061.28935.93Hebei
Beijing3.290.9997.1922154.2Beijing
Benxi0.61.72.3146.1496Liaoning
Binzhou6.722.2534.7392.25Shandong
Changchun0.6420.3525.876751.3Jilin
Changsha0.9360.40616.616815.47Hunan
Changshu19.3272.852122.4106.8Jiangsu
Changzhou9.0552.65428.1472.3Jiangsu
Chaohu10.563.3359.8979.62Anhui
Chengdu0.7630.323119.4231633Sichuan
Chongqing2.5390.2193.2633101.79Chongqing
Dalian0.2760.6392.463595.2Liaoning
Dezhou1.4190.554.7581Shandong
Dongguan2.90424.22739.554839.22Guangdong
Dongying6.8682.18524.365217.21Shandong
Foshan45.55.0253.1790.57Guangdong
Fuzhou0.5330.4623.993774Fujian
Guangzhou5.4414.68120.2151490.44Guangdong
Guiyang0.5780.25817.135488.19Guizhou
Haining27.4122.4549.38685.85Zhejiang
Hangzhou49.9610.8874.145980.6Zhejiang
Harbin0.5580.1578.958951.5Heilongjiang
Hefei1.8530.29320.647808.7Anhui
Heihe0.020.010.54159.3Heilongjiang
Hohhot0.40.1913.663312.6Inner Mongolia
Hong Kong1.045.8688.068745.1Hong Kong
Huaian4.7380.43812.5492.5Jiangsu
Jinan9.6483.84730.564746.04Shandong
Jining1.0350.0452.9834.59Shandong
Jiujiang156.52.17175489.68Jiangxi
Kunming6.7381.11360.39685Yunnan
Lanzhou0.850.212.18374.16Gansu
Lhasa0.750.4514.8555.44Tibet
Lianyungang61.42186.17332.63452Jiangsu
Macao1.36.27.563.16Macao
Mudanjiang0.8201.97252.5Heilongjiang
Nanchang6.8080.9718.593554.55Jiangxi
Nanjing8.781.08327.824843.62Jiangsu
Nanning0.10.25364.093725.41Guangxi
Shanghai12.4512.34526.9132423.78Shanghai
Shenyang1.6410.8965.989831.6Liaoning
Shenzhen8.5489.95423.9021302.66Guangdong
Shijiazhuang17.6019.92145.7661095.16Hebei
Suzhou34.6462.09561.3131072.17Jiangsu
Taian1.3570.3927.2564Shandong
Taipei3.75.49.1262.6Taiwan
Taiyuan0.740.4930.697442.15Shanxi
Tianjin2.2321.0187.6041559.6Tianjin
Urumqi0.40.1959.64350.58Xinjiang
Wuhan1.8970.21213.7711108.1Hubei
Wuxi27.6986.85493.622657.45Zhejiang
Xiamen2.60.763.36411Fujian
Xi'an0.3820.4969.5621000.37Shaanxi
Xining0.4450.21511.39237.11Qinghai
Xuzhou6.8330.87512.5880.2Jiangsu
Yangzhou7.6510.97112.5453.1Jiangsu
Yinchuan0.040.46828.036225.06Ningxia
Zhengzhou0.5160.18315.4761013.6Henan
Zhenjiang4.6741.6816319.64Jiangsu
Zhongshan0.60.61.2331Guangdong
Zhuhai0.7651.312.075189.11Guangdong
Zhuzhou2.110.335.54402.08Hunan
Zibo15.892.1141.01470.2Shandong
Zigong467.9551.606502.926292Sichuan

不同國家和美國各州發布的飲用水中 PFAS 準則

DateAuthorityAgency指導值(ng/L)Guideline TypeReference
2006GermanyGerman Ministry of HealthPFOA+PFOS: 300Health-based guide valuehttps://www.umweltbundesamt.de/sites/default/files/medien/pdfs/pft-in-drinking-water.pdf
2006GermanyGerman Ministry of HealthPFOA+PFOS: 100Health-based precautionary valuehttps://www.umweltbundesamt.de/sites/default/files/medien/pdfs/pft-in-drinking-water.pdf
2014ItalyNational Institute of HealthPFOA: 500; PFOS: 30Health-based levelhttp://www.euro.who.int/__data/assets/pdf_file/0018/340704/FINAL_pfas-report-20170530-h1200.pdf
2015DenmarkDanish Environmental Protection AgencyPFOS: 100; PFOA: 100Quality Criteriahttp://mst.dk/service/publikationer/publikationsarkiv/2015/apr/perfluoroalkylated-substances-pfoa-pfos-and-pfosa/
2016AustraliaDepartment of HealthPFOA: 560; PFOS+PFHxS: 70Health-based guidancehttp://www.health.gov.au/internet/main/publishing.nsf/content/ohp-pfas-hbgv.htm
2016USAUSEPAPFOS+PFOA: 70Lifetime Health Advisoryhttps://www.epa.gov/sites/production/files/2016-05/documents/pfos_health_advisory_final_508.pdf
2016ConnecticutDepartment of Public HealthPFOA+PFOS+PFHxS+PFNA+PFHpA: 70Action Levelhttp://portal.ct.gov/DPH/Search-Results?SearchKeyword=PFAS
2018European UnionEuropean CommissionIndividual PFASs: 100Proposed Drinking Water Directiveec.europa.eu/environment/water/water-drink/review_en.html
2018European UnionEuropean CommissionSum of PFASs: 500Proposed Drinking Water Directiveec.europa.eu/environment/water/water-drink/review_en.html
2018CanadaHealth CanadaPFOA: 200; PFOS: 600Drinking Water Screening Valuehttps://www.canada.ca/en/services/health/publications/healthy-living/water-talk-drinking-water-screening-values-perfluoroalkylated-substances.html
2019MinnesotaDepartment of HealthPFOA: 35; PFOS: 15; PFHxS: 47; PFBA: 7000; PFBS: 2000Health Based Guidance for Waterhttps://www.health.state.mn.us/communities/environment/hazardous/topics/pfcs.html#safelevels
2019California Water Resources Control BoardPFOA: 5.1; PFOS: 6.5Notification Levelshttps://www.ecos.org/news-and-updates/california-lowers-pfas-notification-levels-initiates-regulatory-process/
2019VermontDepartment of Environmental ConservationPFOA+PFOS+PFHxS+PFHpA+PFNA: 20Maximum Contaminant Levelshttps://dec.vermont.gov/water/drinking-water/water-quality-monitoring/pfas
2020MichiganDepartment of Environment, Great Lakes, and EnergyPFOA: 8; PFNA: 6; PFHxA: 400000; PFOS: 16; PFHxS: 51; PFBS: 420; HFPO-DA: 370Maximum Contaminant Levelshttps://www.michigan.gov/egle/0,9429,7-135--534660--,00.html
2020New YorkDepartment of HealthPFOA: 10; PFOS: 10Maximum Contaminant Levelshttps://www.wamc.org/post/nys-adopts-drinking-water-standards-three-emerging-contaminants
2020New HampshireDepartment of Environmental Services PFOA: 12; PFOS: 15; PFHxS: 18; PFNA: 11 Maximum Contaminant Levelshttps://www4.des.state.nh.us/nh-pfas-investigation/?p=1185
2020New JerseyDepartment of Environmental ProtectionPFOA: 13; PFOS: 14Maximum Contaminant Levelshttps://www.penn-er.com/2020/06/08/new-jersey-establishes-mcls-ground-water-and-remediation-standards-for-pfoa-pfos/
2020MassachusettsDepartment of Environmental ProtectionPFOA+PFOS+PFHxS+PFNA+PFHpA+PFDA: 20Maximum Contaminant Levelshttps://www.mass.gov/lists/development-of-a-pfas-drinking-water-standard-mcl
2020SwedenSwedish Food AgencyPFOA+PFOS+PFHxS+PFNA+PFHpA+PFDA+PFBS+6:2FTS+PFBA+PFPeA+PFHxA: 90Action Levelhttps://www.livsmedelsverket.se/en/food-and-content/oonskade-amnen/miljogifter/pfas-in-drinking-water-fish-risk-management

數據來源及完整論文請參閱:Per- and polyfluoroalkyl substances (PFASs) in Chinese drinking water: risk assessment and geographical distribution。Liu, L., Qu, Y., Huang, J. et al. Per- and polyfluoroalkyl substances (PFASs) in Chinese drinking water: risk assessment and geographical distribution. Environ Sci Eur 33, 6 (2021). https://doi.org/10.1186/s12302-020-00425-3

如何在 WordPress 中展示上面的數據圖表

這裏使用了開源的js插件 DataTables 來展示上面的圖表。安裝方法可以參考官方安裝文檔,非常簡單。考慮到文件裏推薦的 CDN 方法在中國可能水土不服,所以我使用本地的安裝方法。在下載頁面勾選需要的組件(我只選了必須的 css 和 js,由於 WordPress 已自帶 jquery,所以不用選jquery)然後下載到一個 zip 包。我是將其上傳到 WordPress 的 wp-content/uploads/share/js 目錄中,然後解壓並修改文件所有者爲nginx( chown nginx:nginx -R DataTables ),就可以使用了。文章裏用到的代碼如下:

<link rel="stylesheet" type="text/css" href="/wp-content/uploads/share/js/datatables.min.css"/>
<script type="text/javascript" src="/wp-content/uploads/share/js/datatables.min.js"></script>

<table id="table_id" class="display">
    <thead>
        <tr>
            <th>Column 1</th>
            <th>Column 2</th>
        </tr>
    </thead>
    <tbody>
        <tr>
            <td>Row 1 Data 1</td>
            <td>Row 1 Data 2</td>
        </tr>
        <tr>
            <td>Row 2 Data 1</td>
            <td>Row 2 Data 2</td>
        </tr>
    </tbody>
</table>

<script>
jQuery(document).ready(function($) {
    $('#table_id').DataTable();
} );
</script>
分類
网站

本站域名變更:從 wupo 到 shaman

首先感謝 EU.org 提供的 free 二級域名,本站從即日起開始在新域名 https://ft.shaman.eu.org 下更新,原 https://ft.wupo.info 會在一天後跳轉到本站。 https://emanon.wupo.info 由於停更,所以本次不會遷移到新域名,但是會在明年域名到期後下線。Emanon 有一個自然觀察的 Youtube 頻道,裡面都是些常見物種記錄。


本次遷移域名遇到的小問題

由於不是很緊迫,所以我是在同一個服務器上複製了一份數據庫,然後又複製了一份 wordpress 文件夾,最後再複製一份 nginx 的配置文件就搞定了。如果遇到 nginx 500 錯誤,記得檢查文件權限。需要在數據庫里操作的幾處我列一下:

#如果只是修改域名,則只需執行下面兩行
update options set option_value="https://ft.shaman.eu.org" where option_name="siteurl";
update options set option_value="https://ft.shaman.eu.org" where option_name="home";

#如果要修改用戶的電子郵箱
update users set user_email='[email protected]' where user_login='user1';
#如果要修改管理員的電子郵箱
update options set option_value='[email protected]' where option_name='admin_email';

還有一個問題是,配置 cloudflare 時出現循環 301 Moved Permanently 跳轉。原因是 cloudflare 默認的 SSL/TLS 設置是 Flexible,這種模式意味着用戶到 cloudflare 是 https,而 cloudflare 到我的服務器卻是 http。但是我的服務器也配置了如果是 http 就重定向到 https。於是就循環請求了。解決辦法是把 cloudflare的 SSL/TLS 設置為 Full (strict) 模式。

nginx 和 cloudflare 設置 301 跳轉都很簡單,這裡寫下用 cloudflare 設置跳轉的方法。在 Page Rule 中新建一條(免費用戶最多可以創建三條)Page Rule。URL matches 填「*ft.wupo.info/*」,接着的兩個下拉框分別選「Forwarding URL」和「301 - Permanent Redirect」,最後一個輸入框填「https://ft.shaman.eu.org/$2」。保存並啟用就可以了。

本文更新於 2020/11/04。

分類
网站

嘗試攔截來自 Cloudflare IP 的密碼猜解

我的 WordPress 安裝了 Limit Login Attempts 插件,其中可以設置輸入錯誤密碼 m 次封鎖這個 IP n 小時,並記錄下這個 IP。本站最近有很多來自 Cloudflare IP 的錯誤登錄嘗試,於是我試圖使用下面兩個方法攔截,可惜結果證明無效。於是我使用了 OTP (One-time password)來增加暴力破解的難度,直到再次找到可行的攔截方法。我還安裝了一個修改 WordPress 登錄頁網址的插件 WPS Hide Login,應該也能有些作用。

Cloudflare Firewall

免費 Cloudflare 用戶可以創建 5 個防火牆規則,我建立了如下兩條:

(http.host in {"ft.wupo.info"} and ip.geoip.asnum in {13335})
(ip.src in {173.245.48.0/20 103.21.244.0/22 103.22.200.0/22 103.31.4.0/22 141.101.64.0/18 108.162.192.0/18 190.93.240.0/20 188.114.96.0/20 197.234.240.0/22 198.41.128.0/17 162.158.0.0/15 104.16.0.0/12 172.64.0.0/13 131.0.72.0/22} and http.host eq "ft.wupo.info")

我隨便查了幾個 Cloudflare IP 的 AS 編號都是 13335,所以第一條就是如果 IP 的 AS 編碼是 13335 就攔截。順便推薦個我經常用的查 IP 的網站 IP-API

第二條是根據 IP 地址來封鎖,從這裡獲得 Cloudflare 的 IP

(ip.geoip.country ne "KP" and http.host eq "ft.wupo.info")

如果想根據 IP 所在國家來限制訪問也是可以的,上面的便是限制了北韓以外 IP 的訪問本站。

在 Limit Login Attempts 插件中限制 IP

這個就是一行一個填已經攔截的 IP 就可以了,可以把所有 IP 複製下來,然後粘貼到 LibreOffice Calc 中,按需要進行排序後複製出 IP 列貼回插件中就可以了。