利用爬虫将电影网站打包成一个APP【三】

q3196355 · 发表于 2019-8-22 10:55:33

第二大步:
然后我们来做详情页
通过Intent获取到上个页面传递过来的URL,图片链接和标题,通过URL获取整个页面数据,然后正则匹配,其他需要的数据

利用爬虫将电影网站打包成一个APP【三】图片ID:984

RequestQueue queue = Volley.newRequestQueue(this);
   MyStringRequest stringRequest = new MyStringRequest(mRequestUrl, new Response.Listener<String>() {
         @Override
         public void onResponse(String response) {
            //Log.e(TAG, response );
            String regEx = "<li><a href=\"/(.+?)\" >(.+?)</a></li>";
            /*
            <div class="movie"><ul><div class="img"><div class="img-box-2"></div><img src="http://88.meenke.com/img_buyhi/201805/2018052876045761.jpg" alt="帝王攻略" border="0"></div><h1>帝王攻略</h1><li>更新至：[17]</li><li>年　代：2018</li><li>类　型：<a href="/dhp_lianzai/Index.html" target="_blank">动画连载</a></li><li class="cksc"><a id="shoucang" href="#sc">收藏</a></li></ul></div>
            */

            Pattern pattern = Pattern.compile(regEx);
            mMatcher = pattern.matcher(response);
            new Thread(new Runnable() {
               @Override
               public void run() {
                     while (mMatcher.find()) {
                        //Toast.makeText(DetailsActivity.this, mMatcher.group() + "", Toast.LENGTH_SHORT).show();
                        //Log.e(TAG, mMatcher.group());
                        //Log.e(TAG, mMatcher.group(1));
                        //Log.e(TAG, mMatcher.group(2));
                        String url = mMatcher.group(1);
                        String title = mMatcher.group(2);
                        //Log.e(TAG, mMatcher.group(3));
                        //Log.e(TAG, mMatcher.group(4));
                        DetailsBean dataBean = new DetailsBean();
                        dataBean.setTitle(title);
                        dataBean.setUrl(url);
                        mData.add(dataBean);
                     }
               }
            }).start();
            DetailAdapter adapter = new DetailAdapter(DetailsActivity.this,mData,mGridLayoutManager);
            mDaRecyView.setAdapter(adapter);
            adapter.setItemClickListener(new DetailAdapter.OnItemClickListener() {
               @Override
               public void onItemClick(int position) {
                  String url = "http://m.yiybb.com/"+mData.get(position).getUrl();
                  Intent intent = new Intent(DetailsActivity.this,PlayActivity.class);
                  intent.putExtra("url",url);
                  startActivity(intent);
               }
            });
         }
   }, new Response.ErrorListener() {
         @Override
         public void onErrorResponse(VolleyError error) {
            Log.e(TAG, error.getMessage(), error);
            Toast.makeText(DetailsActivity.this, "网络不稳定!", Toast.LENGTH_SHORT).show();

         }
   });
   queue.add(stringRequest);
}

第三大步:
最重要就是播放页面了,这里费了点时间
刚开始我也是按照前2步骤去操作,但是发现怎么都爬不到获取不到链接打开原网站才发现播放链接是动态加载的而且还3层加密了!MMP
打开播放页面,F12选择sources,查看源码

利用爬虫将电影网站打包成一个APP【三】图片ID:2352

<div class="playing">就是播放页面,但是里面空的,证明这是动态加载了这个标签
继续看源码找script标签看下里面做了什么
<script language="javascript">var StrHtml;var url=set_code(unescape("JN0HT%250G%256B%256BJeN.Ty6l.167%256Bx%256BlDvoVW%256B1dTw8xE.mkyw",0,0));var nexturl="no";var nextpath="no";var Player={Url:url,Height:240,Width:600,Show:null};function $ShowPlayer(w,h){document.write($Showhtml());}</script>
<script language="javascript" src="Play/23.js"></script><script language="javascript">$ShowPlayer(600,240);</script>
JN0HT%250G%256B%256BJeN.Ty6l.167%256Bx%256BlDvoVW%256B1dTw8xE.mkyw是第一加密后的链接,至于怎么来的,我们不管他,这个数据是可以爬出来的
unescape()方法是JS自带解密方法
set_code()再次加密

找到了主要的逻辑,顺着思路往下走,
在netWork刷新页面看下加载了那些东西,主要看js,发现加载了这么几个东东

利用爬虫将电影网站打包成一个APP【三】图片ID:5588

一个个查看吧,里面搜索set_code这个方法

利用爬虫将电影网站打包成一个APP【三】图片ID:2401

找到了set_code方法:
大概看了下,里面又调用了____e() 和 ____d()
以为这样就找到了链接,我刚开始也这么觉得,但是在网页上打开还是错误的
继续回到sources,查看源码

<script language="javascript" src="Play/23.js"></script><script language="javascript">$ShowPlayer(600,240);</script>
这个才是主要动态加载标签的主要逻辑,在源码了找到Play/23.js
function $ShowPlayer(width,height){
      StrHtml = '<iframe id="ffplayer" src="/ck/ck.html?'+url+'|" width="96%" height="94%"  allowfullscreen="true" frameborder="no" border="0" marginwidth="0" marginheight="0" scrolling="no"></iframe>';
      document.write(StrHtml);
}

将之前加密的URL再一次拼接!
找到了加密方法,那我们来做一个工具类将其解密

<!DOCTYPE html>
<html>

      <head>
            <meta charset="utf-8" />
            <title></title>
      </head>

      <body>
            <script language="JavaScript">
                     var StrHtml;
                     function ____e(){
                              return"0123456789,ABCDEFG,HIJKLMN,OPQRST,UVWXYZ,abcdefg,hijklmn,opqrst,uvwxyz"
                              }
                     function ____d(){
                              return"4560123987,GFEDCBA,MHIJLNK,PQRSTO,ZUVWXY,gfedcba,mhijlnk,pqrsto,zuvwxy"
                              }
                     function set_code(s,en,isN){
                              var e_s = en?____e():____d(), d_s = en?____d():____e(),str="";
                              e_s=isN?e_s.split(",")[0]:e_s,d_s=isN?d_s.split(",")[0]:d_s;
                              for(var i=0;i<s.length;i++){
                                    n=-1;n=e_s.indexOf(s.charAt(i));
                                    if(n!=-1){
                                             str+=d_s.charAt(n)
                                    }else{
                                             str+=s.charAt(i)
                                    }
                              }
                              return str
                     };

                     function getUrl(first_url){
                              var url = set_code(unescape(first_url, 0, 0));
                              StrHtml="http://m.yiybb.com/ck/ck.html?" +url+ "|";
                              window.open(StrHtml);
                     }

            </script>

               </body>

</html>

接下来将HTML放到asset/index.html

mWebView.loadUrl("file:///android_asset/index.html");
      WebSettings settings = mWebView.getSettings();
      settings.setJavaScriptEnabled(true);
      mWebView.addJavascriptInterface(this, "android");
      mWebView.setWebChromeClient(webChromeClient);
      mWebView.setWebViewClient(mWebViewClient);
      settings.setDomStorageEnabled(true);

这样就可以通过webView与本地的HTML文件交互,
接下来获取到播放页面第一次加密的链接,

RequestQueue queue = Volley.newRequestQueue(this);
   MyStringRequest stringRequest = new MyStringRequest(mUrl, new Response.Listener<String>() {
         @RequiresApi(api = Build.VERSION_CODES.KITKAT)
         @Override
         public void onResponse(String response) {
            Log.d(TAG, response);
            String regEx = "unescape\$\"(.+?)\",0,0\$";
            Pattern pattern = Pattern.compile(regEx);
            mMatcher = pattern.matcher(response);

            new Thread(new Runnable() {
               @Override
               public void run() {
                     while (mMatcher.find()) {
                        Log.e(TAG, mMatcher.group(1));
                        mFirst_url = mMatcher.group(1);
                        final String requestUrl = "javascript:getUrl('"+mFirst_url+"')";
                        Log.e(TAG, requestUrl );
                        mWebView.post(new Runnable() {
                           @Override
                           public void run() {
                                 mWebView.loadUrl(requestUrl);

                           }
                        });

                     }
               }
            }).start();

         }
   }, new Response.ErrorListener() {
         @Override
         public void onErrorResponse(VolleyError error) {
            Log.e(TAG, error.getMessage(), error);

            Toast.makeText(PlayActivity.this, "网络不稳定!加载失败!请稍后重试!", Toast.LENGTH_SHORT).show();

         }
   });
   queue.add(stringRequest);

}

调用html自己写的工具类getUrl方法

将获取到的参数传进去,然后再加载网页

[Android] 利用爬虫将电影网站打包成一个APP【三】

相关帖子

浏览过的版块